Commit Graph

198 Commits

Author SHA1 Message Date
aoiasd
e107d0794c
support complex delete expression (#25752)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-09-12 10:19:17 +08:00
congqixia
ac45af585b
Make segment loaded successful put in manager even ctx done (#26992)
Leave segment loaded in manager even wait other segment failed
See also #26908
Fix error case in distributed scenario

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-11 21:33:18 +08:00
congqixia
2a5d574a0d
Fix querynodev2 concurrent load logic (#26959)
Fix logic error from #26926
function `waitSegmentLoadDone` shall return error when context is done

Make delegator control concurrency for each same segment
Related to #26908

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-10 07:41:18 +08:00
congqixia
c6116d1819
Remove segment to LocalSegment type assertion (#26931)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-09 10:35:16 +08:00
congqixia
c8f9f22c4a
Fix segment loader return false success (#26926)
`waitSegmentLoadDone` did not check waitCh result is success or failure
after load return without error, delegator will assume all segments are loaded

This PR changes waitCh to loadResult with `sync.Cond` with `atomic.Int32` to represent status

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-08 16:41:16 +08:00
Xu Tong
9166011c4a
Add float16 vector (#25852)
Signed-off-by: Writer-X <1256866856@qq.com>
2023-09-08 10:03:16 +08:00
SimFG
28681276e2
Improve the retry of the rpc client (#26795)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-06 17:43:14 +08:00
MrPresent-Han
528948559f
fix false load failure for long unserviable period(#26813) (#26818)
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-09-06 12:57:15 +08:00
congqixia
1a8cf5c415
Organize all mockery generation commands in Makefile (#26826)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-04 21:19:48 +08:00
yah01
3349db4aa7
Refine errors to remove changes breaking design (#26521)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-04 09:57:09 +08:00
XuanYang-cn
ef75784715
Fix LoadSegmentLatency metric p99 (#26761)
See also: #26743

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-01 11:07:07 +08:00
MrPresent-Han
8330c18dc9
add log for loading segment(#26564) (#26640)
/kind improvement

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-08-31 12:03:00 +08:00
yah01
bfcc691129
Fix segment leaked if task canceled (#26685)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-30 14:17:03 +08:00
congqixia
95dcf7fa06
Fix memory leak when Put duplicated segments (#26693)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-30 11:37:00 +08:00
congqixia
1cf6e00fa6
Improve segment manager interface (#26637)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-29 15:46:27 +08:00
Xiaofan
7b3a1dc93f
Fix load index panic (#26457)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-08-21 09:56:20 +08:00
wei liu
3b11ad2695
fix dead lock in concurrent search/release segment (#26435)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-18 15:28:18 +08:00
Xiaofan
ba806429fa
Remove frequent log on search growing segment (#26315)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-08-17 15:44:53 +08:00
yah01
5dae6a654a
Protect segment from being released while query/search (#26322)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-16 18:38:17 +08:00
yah01
3d8871b30e
Fix data race while updating segment version (#26387)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-16 16:30:18 +08:00
congqixia
ec65a4e048
Use float64 for resource estimation log (#26335)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-15 14:25:33 +08:00
Enwei Jiao
7d61355ab0
Refactor log for Query (#26310)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-08-14 18:57:32 +08:00
yah01
a173486d2e
Fix calculation of memory usage prediction for mmap mode (#26264)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-12 17:19:31 +08:00
yah01
48422dd4c5
Fix spawn too many threads (#26293)
- Low the thread pool cap
- Limit CGO calls concurrency

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-11 18:29:29 +08:00
Gao
b6fcbb0998
Support ScaNN index (#26099)
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-08-11 14:21:29 +08:00
wei liu
b47a72bfcf
fix set dirty segment distribution to leader view (#26180)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-11 11:21:32 +08:00
xige-16
1971d98897
Add disk metric info (#25675)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-08-11 10:35:42 +08:00
Enwei Jiao
ca1349708b
Remove time travel ralted testcase (#26119)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-08-10 18:53:17 +08:00
xige-16
1055c90456
Add default retrieve limit (#24782)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-08-10 14:11:15 +08:00
yah01
300fef446b
Enable mmap for vector index (#25877)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-10 13:59:15 +08:00
yah01
889424b3f9
Fix load index with empty file list (#26236)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-09 18:39:16 +08:00
wei liu
f33a89387f
enable config shard level cost metrics whether contains worker's cost (#26132)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-09 11:37:15 +08:00
MrPresent-Han
68ecf49ed5
fix lost parameter for threadCoefficient(#25781) (#26109)
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-08-03 12:05:06 +08:00
yah01
d9074bcb06
Remove unused code (#26100)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-08-03 09:45:05 +08:00
yah01
9c55a7f422
Add worker num as one of load resource (#26045)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-01 21:47:06 +08:00
Jiquan Long
5c1f79dc54
Push down the limit operator to segcore (#25959)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-08-01 20:29:05 +08:00
aoiasd
7fec0d61cc
Refine test chunkmanager of querynode v2 (#25888)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-08-01 10:33:04 +08:00
congqixia
8b5e276361
Use different cgo pool for SQ and other operations (#26021)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-01 09:19:05 +08:00
yah01
39b00b97a6
Add more logs for committed resources (#26026)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-31 19:27:03 +08:00
yah01
8245e078c0
Add max segment size log (#26015)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-31 14:37:04 +08:00
congqixia
5fbe6e99fd
Remove debug log from segment loader (#25937)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-26 18:51:01 +08:00
wei liu
302897f866
refine look aside balance logic (#25837)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-25 18:51:01 +08:00
yah01
3735a097e6
Refine the memeory usage prediction (#25889)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-25 17:41:01 +08:00
congqixia
0bc03ede0d
Add eventlog pkg and support grpc streaming event observation (#25812)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-25 17:23:01 +08:00
yah01
168f346620
Remove deprecated interface (#25864)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-07-24 17:47:00 +08:00
Cai Yudong
9a4761dcc7
Remove binary metrics TANIMOTO/SUPERSTRUCTURE/SUBSTRUCTURE (#25708)
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-07-19 16:16:58 +08:00
yah01
079cd9dc70
Add caller skip to improve the log for segcore error (#25710)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-18 19:54:57 +08:00
congqixia
cc71e065e3
Support load index after segment loaded (#25567)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-18 10:51:19 +08:00
congqixia
c490b30db7
Fix segment loader skip wait if all segments are loading (#25643)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-17 19:51:18 +08:00
congqixia
a7e6f08183
Fix DeleteCollection not protected (#25599)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-14 17:04:31 +08:00
yah01
d216f9abda
Clear collection meta after all channels/segments released (#25486)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-14 10:28:30 +08:00
yah01
205a7c430a
Fix QueryNode panic while inserting to released segment (#25495)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-12 19:48:29 +08:00
cai.zhang
81278f4aea
Fix bug of missing JSON type when sorting retrieve results (#25412) (#25455)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-07-10 20:02:28 +08:00
yah01
dd5f896dc8
Load batch by batch (#25212)
This will significantly reduce the memory usage while loading
- 1x memory usage and MBs overhead for buffer (memory mode)
- only MBs overhead for buffer (mmap mode)

Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-06 13:58:27 +08:00
congqixia
efdd71c640
Make cgo pool size larger than worker pool size (#25318)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-05 16:56:26 +08:00
Enwei Jiao
816158e4af
Remove outdated searchplan (#25282)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-07-04 18:30:25 +08:00
yihao.dai
c56a26a556
Add SetMetricType and set metric only for shard leader (#25285)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-07-04 10:20:25 +08:00
yah01
7b5013664d
Support graceful stop (#25226)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-30 17:28:23 +08:00
Jiquan Long
8d193a3e56
Skip unnecessary reduce phase during search (#25166)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-06-27 19:22:45 +08:00
yihao.dai
29b9b26de8
Improve get vector error msg (#24838)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-06-27 16:34:45 +08:00
xige-16
33c2012675
Add more metrics (#25081)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-06-26 17:52:44 +08:00
jaime
18df2ba6fd
[Cherry-Pick] Support Database (#24769)
Support Database(#23742)
Fix db nonexists error for FlushAll (#24222)
Fix check collection limits fails (#24235)
backward compatibility with empty DB name (#24317)
Fix GetFlushAllState with DB (#24347)
Remove db from global meta cache after drop database (#24474)
Fix db name is empty for describe collection response (#24603)
Add RBAC for Database API (#24653)
Fix miss load the same name collection during recover stage (#24941)

RBAC supports Database validation (#23609)
Fix to list grant with db return empty (#23922)
Optimize PrivilegeAll permission check (#23972)
Add the default db value for the rbac request (#24307)

Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-06-25 17:20:43 +08:00
xige-16
04082b3de2
Migrate the ability to upload and download binlog to cpp (#22984)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-06-25 14:38:44 +08:00
yah01
a413842e38
Fix deleted data is still visible (#24849)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-16 17:16:41 +08:00
wei liu
e40d95e093
add search/query request cost metrics on qn (#24413)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-06-15 14:24:38 +08:00
congqixia
41af0a98fa
Use go-api/v2 for milvus-proto (#24770)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
Enwei Jiao
d3af451d92
Upgrade golangci-lint (#24707)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-06-07 19:34:36 +08:00
yah01
fb66924c90
Set metric type while init the collection (#24665)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-06 13:24:36 +08:00
yihao.dai
b0ce4cff83
Assign metric type in search request when user has not explicitly assigned (#24591)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-06-01 18:10:32 +08:00
congqixia
31880ab427
Use conc.PoolOption instead of ants.Option (#24585)
- Add conc.PoolOption to setup conc.Pool
- Change panic default behavior
- Make future has error when job panicks

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-01 15:52:31 +08:00
congqixia
48dff300db
Fix Get segment returns ambiguous result (#24505)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-30 12:25:28 +08:00
aoiasd
c84bdcea49
merge stats log when segment flushing or compacting (#23570)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-05-29 10:21:28 +08:00
congqixia
165734de4e
Add unit test for invoking method after segment released (#24414)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-26 10:03:27 +08:00
yihao.dai
014387fd94
Forbid to get quantized vector from ChunkManager (#24334)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-24 23:03:27 +08:00
Bingyi Sun
0c8045d66c
Fix crash caused by segment function concurrency (#24351)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-05-24 19:23:27 +08:00
yihao.dai
45aa9779e0
Retrieve segments concurrently (#24245)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-19 18:19:24 +08:00
yihao.dai
1a3dca9b5e
Fix dynamic partitions loading (#24112)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-18 09:17:23 +08:00
MrPresent-Han
b74770b636
support try-best query reducer methods(#24075) (#24103)
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-05-17 16:23:23 +08:00
congqixia
73a181d226
Fix get vector it timeout and improve some string const usage (#24141)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-16 17:41:22 +08:00
yah01
400364483d
Improve predict the resource usage (#23968)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-05-11 15:33:24 +08:00
yihao.dai
7e0d1492c7
Load delete from channel checkpoint (#23961)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-09 19:10:41 +08:00
congqixia
884f7904a1
Re-calculate segment entry num when binlog field is zero (#23883)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-06 14:24:39 +08:00
yihao.dai
02734d9eb0
Fix ignoring error when loading segment (#23830)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-04 15:10:39 +08:00
Gao
e467e0d451
Seperate reading build and load params for diskann (#23758)
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-04-27 20:30:35 +08:00
foxspy
6f4ed517de
add growing segment index (#23615)
Signed-off-by: xianliang <xianliang.li@zilliz.com>
2023-04-26 10:14:41 +08:00
yihao.dai
092d743917
Add support for getting vectors by ids (#23450)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-04-23 09:00:32 +08:00
yah01
546080dcdd
Support to retrieve json (#23563)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-04-21 11:46:32 +08:00
Enwei Jiao
967a97b9bd
Support json & array types (#23408)
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
2023-04-20 11:32:31 +08:00
congqixia
4fe363c4b2
Add KNN cgo pool (#23526)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-04-19 23:56:31 +08:00
yah01
dc0eb433ae
Add IoPoolSize config (#23360)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-04-13 14:50:28 +08:00
Jiquan Long
ad7f3d4c3e
Support Count(*) on querynodev2 (#23321)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-04-11 15:40:31 +08:00
congqixia
0527d2706c
Fix offline set always nil (#23340)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-04-11 14:42:30 +08:00
jaime
c9d0c157ec
Move some modules from internal to public package (#22572)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
congqixia
424570b2b8
Remove annoy related unit tests (#23229)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-04-05 08:14:28 +08:00
yihao.dai
9cecdc4f4e
Fix querynode panic (#23181)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-04-03 14:18:24 +08:00
Enwei Jiao
50dab2d394
Add tracing for retrieve (#23033)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-03-29 18:10:02 +08:00
yah01
826c1fb850
Support mmap on QueryNodeV2 (#23073)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-28 21:30:05 +08:00
yah01
081572d31c
Refactor QueryNode (#21625)
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
2023-03-27 00:42:00 +08:00