issue: #28781#28329
1. There is no need to call `DescribeCollection`, if the collection's
schema is found in the globalMetaCache
2. did `GetProperties` to check the access to Azure Blob Service while
construct the ChunkManager
Signed-off-by: PowderLi <min.li@zilliz.com>
for now the assert method in segcore could accept a string information,
too many codes don't print the value they assert.
make it happy
related #28811
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
issue: #28831
release old delegator before new delegator update it's distribution may
cause `channel not availble` error
This PR will block release old delgator before new delegator finish
`syncDistribution`
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
- Change flowgraphManager to fgManagerImpl
- Change close to stop
- change execute to controlMemWaterLevel
- Change method name of fgManager for readability
- Add mockery for fgmanager
Issue: #28853
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
return the last error but not combining all errors, to improve
readability and erorr handling
resolve: #28572
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
We have implemented the chunkcache (in cpp) to retrieve vectors, hence
rendering the vectorchunkcache (in golang) obsolete.
issue: https://github.com/milvus-io/milvus/issues/28568
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Assign the 'lgtm' and 'approved' labels following the successful testing of the 'Update Knowhere Commit'
then prow/tide bot will auto merge the PR to master branch.
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
Related to #28628
Compacted segment syncing counter is not set correctly in sync task and
the bf write buffer shall not use compacted segment as candidate when
buffering delta data
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Now segcore load system field info as well, the growing segment
assertion shall not pass with "+ 2" value
This will cause all growing segments load failure
Fix#28801
Related to #28478
See also #28524
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #27678
when interimIndex = true, memory predict should be update with the
memory usage of binlog index build process.
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
the incorrect filter causes all LevelZero segment filtered, so the
deleted entities may be still visible
related: #27349
Signed-off-by: yah01 <yah2er0ne@outlook.com>
This PR:
- Separates compaction scheduler and check results loop So that slow in
check-loop doesn't influence execution.
- Cleans compaction tasks when drop a vchannel so dropped-channel's
compaction tasks won't be checked over and over again.
- Skips meta change when meta's already changed, avoid panic
- Remove not inuse injectDone(bool) parameter
See also: #28628, #28209
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
1. modify test cases about output a non-existed field. issue: #28465
2. add cases about creating collection using different language
3. add test case about inserting a string value to a pk(int64) field
4. add test case about search with expression using double quotes.
issue: #28365
Signed-off-by: nico <cheng.yuan@zilliz.com>
1. Drop a non-existing partition will not raise error now. #28722
2. Search will not raise error now when some row has less elements than
expression index. #28293
Signed-off-by: nico <cheng.yuan@zilliz.com>
test: There are too many test cases for bulkinsert+partition_key. Each
case creates 10 bulkinsert tasks to import a file with 100~200 rows. The
default num_partitions is 64 for partition_key. So, each task will
generate 64 tiny segments. There are 10 cases, each case 10 tasks, each
task 64 tiny segment, totally there are 6400 tiny segments generated.
And all these segment row count is less than 1024, no need to build
index, and take part in compaction. There will be lots of compaction
tasks generated. It costs too much time to process these compaction
tasks. Eventually, some cases are timeout after waiting 5 minutes for
their segments to be ready and cases fail.
Specifying the num_partitions to a small value can avoid this problem.
```
[2023-11-21T03:41:16.187Z] testcases/test_bulk_insert.py::TestBulkInsert::test_partition_key_on_json_file[int_scalar-True-True] PASSED [ 54%]
[2023-11-21T03:41:42.796Z] testcases/test_bulk_insert.py::TestBulkInsert::test_partition_key_on_json_file[int_scalar-False-True] PASSED [ 57%]
[2023-11-21T03:42:04.694Z] testcases/test_bulk_insert.py::TestBulkInsert::test_partition_key_on_json_file[string_scalar-True-True] PASSED [ 60%]
[2023-11-21T03:42:31.205Z] testcases/test_bulk_insert.py::TestBulkInsert::test_partition_key_on_json_file[string_scalar-False-True] PASSED [ 63%]
[2023-11-21T03:43:38.876Z] testcases/test_bulk_insert.py::TestBulkInsert::test_partition_key_on_multi_numpy_files[10-150-13-True] XPASS [ 66%]
[2023-11-21T03:49:00.357Z] testcases/test_bulk_insert.py::TestBulkInsert::test_partition_key_on_multi_numpy_files[10-150-13-False] XFAIL [ 69%]
[2023-11-21T03:53:51.811Z] testcases/test_bulk_insert.py::TestBulkInsert::test_partition_key_on_csv_file[int_scalar-True] FAILED [ 72%]
[2023-11-21T03:58:58.283Z] testcases/test_bulk_insert.py::TestBulkInsert::test_partition_key_on_csv_file[int_scalar-False] FAILED [ 75%]
[2023-11-21T04:02:04.696Z] testcases/test_bulk_insert.py::TestBulkInsert::test_partition_key_on_csv_file[string_scalar-True] PASSED [ 78%]
[2023-11-21T04:02:26.608Z] testcases/test_bulk_insert.py::TestBulkInsert::test_partition_key_on_csv_file[string_scalar-False] PASSED [ 81%]
```
Signed-off-by: yhmo <yihua.mo@zilliz.com>
see also: https://github.com/milvus-io/milvus/issues/28509
Currently Minio latency monitoring for get operation only collects the
duration of getting object (which just returns an io.Reader and does not
really read from minio), this pr will correct this behavior.
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Related to #28736#28748
See also #27675
Previous PR: #28646
This PR fixes `SegmentNotFound` issue when compaction happens multiple
times and the buffer of first generation segment is sync due to stale
policy
Now the `CompactSegments` API of metacache shall update the compactTo
field of segmentInfo if the compactTo segment is also compacted to keep
the bloodline clean
Also, add the `CompactedSegment` SyncPolicy to sync the compacted
segment asap to keep metacache clean
Now the `SyncPolicy` is an interface instead of a function type so that
when it selects some segments to sync, we colud log the reason and
target segment
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
