Commit Graph

354 Commits

Author SHA1 Message Date
Xiaofan
af66a0b621
Fix Time Tick session2ChanTs map use the same nodeID (#20537)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-11-14 14:41:11 +08:00
Enwei Jiao
b37a9c5fe2
Fix log to file not work (#20541)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-14 14:23:07 +08:00
Enwei Jiao
7e56e7a976
Fix datarace at Setlogger (#20376)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-08 14:33:03 +08:00
Enwei Jiao
956c5e1b9d
Make Params singleton (#20088)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-04 14:25:38 +08:00
Jiquan Long
d9081980e2
[skip e2e] Exit migration after session expired (#20282)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-11-03 11:47:36 +08:00
Jiquan Long
f2f10cb114
[skip e2e] Expire migration session asap and fix exit code (#20224)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-11-01 11:47:34 +08:00
Jiquan Long
e9cd2cb42a
[skip e2e] Fix empty index build meta migration (#20206)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-10-31 17:33:35 +08:00
Jiquan Long
3625b02d50
[skip e2e] Clean load meta of 2.2 when rollback (#20195)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-10-31 13:01:33 +08:00
Jiquan Long
977947e224
[skip e2e] Print detailed migration info (#20178)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-10-28 21:49:33 +08:00
cai.zhang
53289a63e6
Specification log (#20119)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-10-28 17:01:32 +08:00
cai.zhang
1be4d1e267
Fill the WriteHandoff field in the upgrade script (#20139)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-10-27 20:25:35 +08:00
cai.zhang
4848edda8d
Append field indexID for index name (#20046)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-10-27 20:23:38 +08:00
yah01
e54b6181b6
Ignore the collections without index from 2.1 (#20064)
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-25 17:45:31 +08:00
cai.zhang
a60db370e6
Compatibility with handling index params (#19997)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-10-25 16:59:30 +08:00
yah01
aa6b8af80d
Migrate metadata of QueryCoord from 2.1 to 2.2 (#19986)
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-24 23:41:35 +08:00
jaime
b79687687d
Replace segment index file path with index file id within segment index meta (#19866)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-19 16:55:27 +08:00
Jiquan Long
34e04c27ad
[skip e2e] Fix missing num_rows after meta migration (#19841)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-10-17 19:40:51 +08:00
Jiquan Long
b0a12116c8
Fix meta migration tool (#19814)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-10-17 15:07:25 +08:00
SimFG
a55f739608
Separate public proto files (#19782)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
xige-16
a1db9038fb
Move disk index params to config file (#19714)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-14 17:51:24 +08:00
Enwei Jiao
82dd1bba1c
Refactor healthz API (#19747)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-10-14 14:19:24 +08:00
Enwei Jiao
4df3552f74
Prepare http admin API (#19676)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-10-13 10:37:23 +08:00
Jiquan Long
a8a074162f
Add meta migration tool (#19709)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-10-12 11:37:23 +08:00
yah01
2925d8a1ac
Remove QueryCoordV1 (#19681)
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-11 11:11:23 +08:00
wayblink
85e04d845c
Move GetComponentStates to milvus.proto, make it an external interface (#19594)
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-10-10 15:55:22 +08:00
congqixia
838a633584
Add ctx parameter for ChunkManager methods (#19546)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-29 16:18:56 +08:00
Xiaofan
928a213e31
Fix Chunk Manager Path mis use (#19141)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-09-23 14:40:51 +08:00
xige-16
0ba47bd0d0
Mix start with more role (#19251)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-09-16 20:38:49 +08:00
SimFG
d7f38a803d
Separate some proto files (#19218)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
jaime
816967c000
Add metrics for etcd (#18962)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-09-02 17:46:59 +08:00
Zach
0c9a10e8f8
1. refine logging interfaces (#18692)
2. adjust logs for query/search requests

Signed-off-by: Zach41 <zongmei.zhang@zilliz.com>

Signed-off-by: Zach41 <zongmei.zhang@zilliz.com>
2022-08-23 10:44:52 +08:00
codeman
6c3dbf0a07
Support meta storing on database (#17236) (#18541)
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
2022-08-11 12:12:38 +08:00
Xiaofan
b427450593
Remove Load Delta Channel Request (#18497)
Fix #18393

remove watch delta channel task related logic

Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2022-08-04 17:50:37 +08:00
jaime
6e7831470c
Revert settings of Pulsar address and refine config code (#18494)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-08-02 21:26:33 +08:00
SimFG
eb1f0bc805
Add the consistency check tool (#17136)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-06-21 10:28:12 +08:00
bigsheeper
c77bc81057
Set jemalloc LD_PRELOAD env to linux image (#17494)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-10 21:58:08 +08:00
shaoyue
7527285ec4
Fix mixture (#17332)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-06-02 10:34:03 +08:00
Ten Thousand Leaves
6f75d02c65
Disable knowhere logging for embedded Milvus and some other tweaks (#16496)
/kind enhancement

issue: #15711
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-20 17:23:46 +08:00
Ten Thousand Leaves
cf9b9aba79
Fix a bug where RPC calls before proxy finishes initialization crashes Milvus (#16474)
/kind bug

issue: #16473
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-14 20:01:34 +08:00
jaime
ed3c29eae7
Support Kafka (#16149)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-04-12 19:47:33 +08:00
godchen
bb7a0766fe
Add dependency factory (#16204)
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-04-07 22:05:32 +08:00
xiyichan
ca129d4308
Support configurable msgstream (#16131)
Signed-off-by: xiyichan <2863768433@qq.com>
2022-03-24 10:15:25 +08:00
Xiaofan
787d4f8bfd
Refine prometheus metrics and etcd log (#16084)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-03-18 15:51:22 +08:00
Ji Bin
3cd28420f1
Support compile under windows (#15786)
This patch makes compile milvus under windows(MSYS), including:
- some cpp adaptation for compile under msys/gcc-10.3
- install toolchain scripts for setup from MinGW/MSYS `scripts/install_deps_msys.sh`
- adaptation for POSIX API use in golang
  * using gofrs/flock instead of syscall.Flock
  * using x/exp/mmap instead of syscall.Mmap
- introducing github actions for build milvus.exe under windows/MSYS
- rocksdb's patch for MSYS
- adaptation for compile knowhere under windows
- a windows package script for pack zip file, `scripts/package_windows.sh`

issue #7706

Signed-off-by: Ji Bin <matrixji@live.com>
2022-03-17 17:17:22 +08:00
Ten Thousand Leaves
a3a29fed06
Add entry point for embedded Milvus. (#16008)
/kind feature

issue: #15711
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-03-16 16:51:22 +08:00
Ten Thousand Leaves
d5bae6710f
Make all logs configurable for standalone/embedded Milvus. (#15926)
Resolves: #15919, #15708
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-03-10 10:35:59 +08:00
jaime
7a3a721380
Reconstruct mqstream module (#15784)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-03-03 21:57:56 +08:00
Nemo
d1b8c9850b
Fix a bug for standalone where it appears that DEPLOY_MODE is get/used before set. (#15712)
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-02-24 14:41:52 +08:00
Xiaofan
be42627ab2
Move Logutil to Util dir (#15447)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-02-09 18:55:46 +08:00
Cai Yudong
1f337891e1
Rename base_param to service_param, rename global_param to component_param (#15463)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-08 20:57:47 +08:00
Cai Yudong
f1ea9613ae
Add EtcdConfig and move PulsarConfig/RocksdbConfig/MinioConfig into BaseParamTable (#15434)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-07 10:09:45 +08:00
Cai Yudong
72a17339f5
[skip e2e] Remove SIGSEGV handling (#15341)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-21 20:03:38 +08:00
Cai Yudong
7b48dc7b84
[skip e2e] Add SIGSEGV handling (#15248)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-17 17:35:35 +08:00
congqixia
3cf44cb83f
Add automaxprocs (#14496)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-29 17:40:59 +08:00
Xiaofan
69087ff8dd
Support embedded etcd (#14333)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-12-29 14:35:21 +08:00
Cai Yudong
d34bb26414
Set rolname when init GlobalParamTable (#14422)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-28 23:12:46 +08:00
Bingyi Sun
02e568603b
Add log with context in DataCoord (#14431)
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-12-28 19:42:19 +08:00
Cai Yudong
891c202b73
Use GlobalParamTable for all components (#13885)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-23 18:39:11 +08:00
zhenshan.cao
c5d91af794
Enable pprof service (#13979)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-12-23 11:39:10 +08:00
congqixia
1dcd06cc78
Add log file info for all kind of segment binlogs (#13606)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-19 20:00:42 +08:00
Bingyi Sun
37705dfbbb
Add module name to logger in proxy (#13580)
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-12-17 19:07:39 +08:00
Cai Yudong
78acb5fe5c
Update main to use role const (#13431)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-15 17:49:09 +08:00
Cai Yudong
024c1ac412
Let standalone share same LogCfg with distributed components (#13332)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-14 18:31:25 +08:00
dragondriver
78698a8918
Add log when healthz check return false (#13261)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-12-14 10:35:11 +08:00
jaime
b43b4e213f
[skip ci] Fix a grammar mistake in roles.go (#12604)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: yun.zhang <yun.zhang@zilliz.com>
2021-12-13 09:29:50 +08:00
congqixia
d68f7602e5
Standalone healthz check include proxy and coordinator state check (#12812)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-07 13:33:43 +08:00
Xiangyu Wang
936b1a7f8b
Remove unused codes (#12780)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-12-06 18:20:31 +08:00
yukun
764cf1a53e
Add rocksmq server close when standalone exit (#12255)
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-12-02 10:27:32 +08:00
dragondriver
b6d0150dc5
Fix health check when disconnect to etcd (#12292)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-11-26 11:39:16 +08:00
dragondriver
6c4c0ef6b5
Add more deploy metrics (#12248)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-11-24 15:43:15 +08:00
XuanYang-cn
7dbb4d10e5
Fix DeltaLogs empty yet appending a deltaInfo bug (#11984)
- Add more logs for startpositions

Fixes: #11970

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-17 17:57:11 +08:00
dragondriver
cc0d6dc6fd
Fix runtime path created by multiple processes (#11966)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-11-17 17:47:12 +08:00
Xiangyu Wang
12f50cb22c
[skip ci]Update OWNERS files (#11898)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-11-16 15:41:11 +08:00
XuanYang-cn
9dcee0ae4d
Fix typo of ected to etcd (#11784)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-15 19:11:52 +08:00
congqixia
413aa42796
Add tool for datacoord segment meta (#11745)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-12 19:01:11 +08:00
Cai Yudong
124bd6c20a
Let param table init once (#11510)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-11-10 10:07:37 +08:00
dragondriver
4dd6051eb2
[skip ci] Fix golint warning of roles.go (#11024)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-11-01 22:35:18 +08:00
groot
fb71fd525a
[skip ci] Typo (#10823)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2021-10-28 18:02:30 +08:00
groot
bad79d9179
[skip ci] Add log (#10822)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2021-10-28 18:00:38 +08:00
dragondriver
807ccd0085
Register healthz router at once when component is created (#9722)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-10-12 19:25:44 +08:00
XuanYang-cn
307d6193d6
[skip ci]Update license in cmd/ (#9508)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-10-09 18:03:34 +08:00
dragondriver
30ecd3164f
[skip ci] Add desc comments for GetComponentStatesInterface (#9506)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-10-08 23:56:13 +08:00
Xiangyu Wang
81073c2f9d
[skip ci]Change name to account in OWNERS file (#9429)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-10-08 09:01:18 +08:00
Xiaofan
2c2b800ff4
Change Log Name with NodeId (#8721)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-10-01 08:52:50 +08:00
XuanYang-cn
c8f48f2e5b
[skip-ci]Fix golint in cmd/components (#8895)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-09-29 23:40:10 +08:00
XuanYang-cn
87b1d66d74
[skip-ci]Fix golint on cmd/components (#8896)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-09-29 23:20:11 +08:00
zhenshan.cao
d1d5f9e409
Replace proto.MarshalTextString with proto.Marshal (#8542)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-09-29 20:26:00 +08:00
XuanYang-cn
0e50fa2d31
[skip-ci]Fix golint in cmd/data_node.go (#8693)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-09-28 22:30:05 +08:00
XuanYang-cn
892991ee61
[skip-ci]Fix golint in cmd data_coord (#8692)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-09-27 19:09:58 +08:00
XuanYang-cn
b92f51bcfe
[skip-ci]Fix golint in cmd/index_node.go (#8694)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-09-27 19:04:08 +08:00
sunby
d5e674cb5c
Change fmt print to log print (#8308)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-09-22 19:48:07 +08:00
dragondriver
85a6953330
Support healthz check api (#8228)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-09-22 16:18:21 +08:00
Xiangyu Wang
77ef692390
Improve import statements in main.go (#8014)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-09-17 11:19:54 +08:00
dragondriver
960741ee16
Support more detailed metrics information (#7472)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-09-03 19:35:47 +08:00
Xiangyu Wang
c62f23e201
[skip ci]Remove neza2017 from reviewers (#7324)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-08-27 00:27:36 +08:00
sunby
4d56d53c0b
Add a tool to move binlog path to SegmentInfo (#7243)
issue: #7214
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-08-24 15:51:51 +08:00
yukun
847586eb95
Add rocksmq retention (#6617)
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-08-16 18:46:10 +08:00
Xiangyu Wang
5e78fd6a97
[skip ci]Update approvers (#7063)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-08-12 19:36:09 +08:00
Cai Yudong
3524d87f9b
optimize stop process logic (#6256)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-07-01 18:13:48 +08:00
yukun
39614aa8eb
Add rocksmq_path in config and paramtable (#6099)
* Add rocksmq_path in config and paramtable

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add rocksdbPath in NewRmsFactory

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Change rdb default path to /vat/lib/milvus/rdb_data

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-06-25 19:44:11 +08:00