aoiasd
e107d0794c
support complex delete expression ( #25752 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-09-12 10:19:17 +08:00
congqixia
ac45af585b
Make segment loaded successful put in manager even ctx done ( #26992 )
...
Leave segment loaded in manager even wait other segment failed
See also #26908
Fix error case in distributed scenario
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-11 21:33:18 +08:00
congqixia
2a5d574a0d
Fix querynodev2 concurrent load logic ( #26959 )
...
Fix logic error from #26926
function `waitSegmentLoadDone` shall return error when context is done
Make delegator control concurrency for each same segment
Related to #26908
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-10 07:41:18 +08:00
congqixia
c6116d1819
Remove segment to LocalSegment type assertion ( #26931 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-09 10:35:16 +08:00
congqixia
c8f9f22c4a
Fix segment loader return false success ( #26926 )
...
`waitSegmentLoadDone` did not check waitCh result is success or failure
after load return without error, delegator will assume all segments are loaded
This PR changes waitCh to loadResult with `sync.Cond` with `atomic.Int32` to represent status
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-08 16:41:16 +08:00
Xu Tong
9166011c4a
Add float16 vector ( #25852 )
...
Signed-off-by: Writer-X <1256866856@qq.com>
2023-09-08 10:03:16 +08:00
SimFG
28681276e2
Improve the retry of the rpc client ( #26795 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-06 17:43:14 +08:00
MrPresent-Han
528948559f
fix false load failure for long unserviable period( #26813 ) ( #26818 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-09-06 12:57:15 +08:00
congqixia
1a8cf5c415
Organize all mockery generation commands in Makefile ( #26826 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-04 21:19:48 +08:00
yah01
3349db4aa7
Refine errors to remove changes breaking design ( #26521 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-04 09:57:09 +08:00
XuanYang-cn
ef75784715
Fix LoadSegmentLatency metric p99 ( #26761 )
...
See also: #26743
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-01 11:07:07 +08:00
MrPresent-Han
8330c18dc9
add log for loading segment( #26564 ) ( #26640 )
...
/kind improvement
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-08-31 12:03:00 +08:00
yah01
bfcc691129
Fix segment leaked if task canceled ( #26685 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-30 14:17:03 +08:00
congqixia
95dcf7fa06
Fix memory leak when Put duplicated segments ( #26693 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-30 11:37:00 +08:00
congqixia
1cf6e00fa6
Improve segment manager interface ( #26637 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-29 15:46:27 +08:00
Xiaofan
7b3a1dc93f
Fix load index panic ( #26457 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-08-21 09:56:20 +08:00
wei liu
3b11ad2695
fix dead lock in concurrent search/release segment ( #26435 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-18 15:28:18 +08:00
Xiaofan
ba806429fa
Remove frequent log on search growing segment ( #26315 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-08-17 15:44:53 +08:00
yah01
5dae6a654a
Protect segment from being released while query/search ( #26322 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-16 18:38:17 +08:00
yah01
3d8871b30e
Fix data race while updating segment version ( #26387 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-16 16:30:18 +08:00
congqixia
ec65a4e048
Use float64 for resource estimation log ( #26335 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-15 14:25:33 +08:00
Enwei Jiao
7d61355ab0
Refactor log for Query ( #26310 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-08-14 18:57:32 +08:00
yah01
a173486d2e
Fix calculation of memory usage prediction for mmap mode ( #26264 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-12 17:19:31 +08:00
yah01
48422dd4c5
Fix spawn too many threads ( #26293 )
...
- Low the thread pool cap
- Limit CGO calls concurrency
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-11 18:29:29 +08:00
Gao
b6fcbb0998
Support ScaNN index ( #26099 )
...
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-08-11 14:21:29 +08:00
wei liu
b47a72bfcf
fix set dirty segment distribution to leader view ( #26180 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-11 11:21:32 +08:00
xige-16
1971d98897
Add disk metric info ( #25675 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-08-11 10:35:42 +08:00
Enwei Jiao
ca1349708b
Remove time travel ralted testcase ( #26119 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-08-10 18:53:17 +08:00
xige-16
1055c90456
Add default retrieve limit ( #24782 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-08-10 14:11:15 +08:00
yah01
300fef446b
Enable mmap for vector index ( #25877 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-10 13:59:15 +08:00
yah01
889424b3f9
Fix load index with empty file list ( #26236 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-09 18:39:16 +08:00
wei liu
f33a89387f
enable config shard level cost metrics whether contains worker's cost ( #26132 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-09 11:37:15 +08:00
MrPresent-Han
68ecf49ed5
fix lost parameter for threadCoefficient( #25781 ) ( #26109 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-08-03 12:05:06 +08:00
yah01
d9074bcb06
Remove unused code ( #26100 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-08-03 09:45:05 +08:00
yah01
9c55a7f422
Add worker num as one of load resource ( #26045 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-01 21:47:06 +08:00
Jiquan Long
5c1f79dc54
Push down the limit operator to segcore ( #25959 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-08-01 20:29:05 +08:00
aoiasd
7fec0d61cc
Refine test chunkmanager of querynode v2 ( #25888 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-08-01 10:33:04 +08:00
congqixia
8b5e276361
Use different cgo pool for SQ and other operations ( #26021 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-01 09:19:05 +08:00
yah01
39b00b97a6
Add more logs for committed resources ( #26026 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-31 19:27:03 +08:00
yah01
8245e078c0
Add max segment size log ( #26015 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-31 14:37:04 +08:00
congqixia
5fbe6e99fd
Remove debug log from segment loader ( #25937 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-26 18:51:01 +08:00
wei liu
302897f866
refine look aside balance logic ( #25837 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-25 18:51:01 +08:00
yah01
3735a097e6
Refine the memeory usage prediction ( #25889 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-25 17:41:01 +08:00
congqixia
0bc03ede0d
Add eventlog pkg and support grpc streaming event observation ( #25812 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-25 17:23:01 +08:00
yah01
168f346620
Remove deprecated interface ( #25864 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-07-24 17:47:00 +08:00
Cai Yudong
9a4761dcc7
Remove binary metrics TANIMOTO/SUPERSTRUCTURE/SUBSTRUCTURE ( #25708 )
...
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-07-19 16:16:58 +08:00
yah01
079cd9dc70
Add caller skip to improve the log for segcore error ( #25710 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-18 19:54:57 +08:00
congqixia
cc71e065e3
Support load index after segment loaded ( #25567 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-18 10:51:19 +08:00
congqixia
c490b30db7
Fix segment loader skip wait if all segments are loading ( #25643 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-17 19:51:18 +08:00
congqixia
a7e6f08183
Fix DeleteCollection not protected ( #25599 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-14 17:04:31 +08:00
yah01
d216f9abda
Clear collection meta after all channels/segments released ( #25486 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-14 10:28:30 +08:00
yah01
205a7c430a
Fix QueryNode panic while inserting to released segment ( #25495 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-12 19:48:29 +08:00
cai.zhang
81278f4aea
Fix bug of missing JSON type when sorting retrieve results ( #25412 ) ( #25455 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-07-10 20:02:28 +08:00
yah01
dd5f896dc8
Load batch by batch ( #25212 )
...
This will significantly reduce the memory usage while loading
- 1x memory usage and MBs overhead for buffer (memory mode)
- only MBs overhead for buffer (mmap mode)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-06 13:58:27 +08:00
congqixia
efdd71c640
Make cgo pool size larger than worker pool size ( #25318 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-05 16:56:26 +08:00
Enwei Jiao
816158e4af
Remove outdated searchplan ( #25282 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-07-04 18:30:25 +08:00
yihao.dai
c56a26a556
Add SetMetricType and set metric only for shard leader ( #25285 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-07-04 10:20:25 +08:00
yah01
7b5013664d
Support graceful stop ( #25226 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-30 17:28:23 +08:00
Jiquan Long
8d193a3e56
Skip unnecessary reduce phase during search ( #25166 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-06-27 19:22:45 +08:00
yihao.dai
29b9b26de8
Improve get vector error msg ( #24838 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-06-27 16:34:45 +08:00
xige-16
33c2012675
Add more metrics ( #25081 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-06-26 17:52:44 +08:00
jaime
18df2ba6fd
[Cherry-Pick] Support Database ( #24769 )
...
Support Database(#23742 )
Fix db nonexists error for FlushAll (#24222 )
Fix check collection limits fails (#24235 )
backward compatibility with empty DB name (#24317 )
Fix GetFlushAllState with DB (#24347 )
Remove db from global meta cache after drop database (#24474 )
Fix db name is empty for describe collection response (#24603 )
Add RBAC for Database API (#24653 )
Fix miss load the same name collection during recover stage (#24941 )
RBAC supports Database validation (#23609 )
Fix to list grant with db return empty (#23922 )
Optimize PrivilegeAll permission check (#23972 )
Add the default db value for the rbac request (#24307 )
Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-06-25 17:20:43 +08:00
xige-16
04082b3de2
Migrate the ability to upload and download binlog to cpp ( #22984 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-06-25 14:38:44 +08:00
yah01
a413842e38
Fix deleted data is still visible ( #24849 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-16 17:16:41 +08:00
wei liu
e40d95e093
add search/query request cost metrics on qn ( #24413 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-06-15 14:24:38 +08:00
congqixia
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
Enwei Jiao
d3af451d92
Upgrade golangci-lint ( #24707 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-06-07 19:34:36 +08:00
yah01
fb66924c90
Set metric type while init the collection ( #24665 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-06 13:24:36 +08:00
yihao.dai
b0ce4cff83
Assign metric type in search request when user has not explicitly assigned ( #24591 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-06-01 18:10:32 +08:00
congqixia
31880ab427
Use conc.PoolOption instead of ants.Option ( #24585 )
...
- Add conc.PoolOption to setup conc.Pool
- Change panic default behavior
- Make future has error when job panicks
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-01 15:52:31 +08:00
congqixia
48dff300db
Fix Get segment returns ambiguous result ( #24505 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-30 12:25:28 +08:00
aoiasd
c84bdcea49
merge stats log when segment flushing or compacting ( #23570 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-05-29 10:21:28 +08:00
congqixia
165734de4e
Add unit test for invoking method after segment released ( #24414 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-26 10:03:27 +08:00
yihao.dai
014387fd94
Forbid to get quantized vector from ChunkManager ( #24334 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-24 23:03:27 +08:00
Bingyi Sun
0c8045d66c
Fix crash caused by segment function concurrency ( #24351 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-05-24 19:23:27 +08:00
yihao.dai
45aa9779e0
Retrieve segments concurrently ( #24245 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-19 18:19:24 +08:00
yihao.dai
1a3dca9b5e
Fix dynamic partitions loading ( #24112 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-18 09:17:23 +08:00
MrPresent-Han
b74770b636
support try-best query reducer methods( #24075 ) ( #24103 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-05-17 16:23:23 +08:00
congqixia
73a181d226
Fix get vector it timeout and improve some string const usage ( #24141 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-16 17:41:22 +08:00
yah01
400364483d
Improve predict the resource usage ( #23968 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-05-11 15:33:24 +08:00
yihao.dai
7e0d1492c7
Load delete from channel checkpoint ( #23961 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-09 19:10:41 +08:00
congqixia
884f7904a1
Re-calculate segment entry num when binlog field is zero ( #23883 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-06 14:24:39 +08:00
yihao.dai
02734d9eb0
Fix ignoring error when loading segment ( #23830 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-04 15:10:39 +08:00
Gao
e467e0d451
Seperate reading build and load params for diskann ( #23758 )
...
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-04-27 20:30:35 +08:00
foxspy
6f4ed517de
add growing segment index ( #23615 )
...
Signed-off-by: xianliang <xianliang.li@zilliz.com>
2023-04-26 10:14:41 +08:00
yihao.dai
092d743917
Add support for getting vectors by ids ( #23450 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-04-23 09:00:32 +08:00
yah01
546080dcdd
Support to retrieve json ( #23563 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-04-21 11:46:32 +08:00
Enwei Jiao
967a97b9bd
Support json & array types ( #23408 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
2023-04-20 11:32:31 +08:00
congqixia
4fe363c4b2
Add KNN cgo pool ( #23526 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-04-19 23:56:31 +08:00
yah01
dc0eb433ae
Add IoPoolSize config ( #23360 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-04-13 14:50:28 +08:00
Jiquan Long
ad7f3d4c3e
Support Count(*) on querynodev2 ( #23321 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-04-11 15:40:31 +08:00
congqixia
0527d2706c
Fix offline set always nil ( #23340 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-04-11 14:42:30 +08:00
jaime
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
congqixia
424570b2b8
Remove annoy related unit tests ( #23229 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-04-05 08:14:28 +08:00
yihao.dai
9cecdc4f4e
Fix querynode panic ( #23181 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-04-03 14:18:24 +08:00
Enwei Jiao
50dab2d394
Add tracing for retrieve ( #23033 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-03-29 18:10:02 +08:00
yah01
826c1fb850
Support mmap on QueryNodeV2 ( #23073 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-28 21:30:05 +08:00
yah01
081572d31c
Refactor QueryNode ( #21625 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
2023-03-27 00:42:00 +08:00