For some time now, GitLab has been working on enabling the Elasticsearch integration on GitLab.com to allow as many GitLab.com users as possible access to the高级全局搜索特征。Last year, after enabling Advanced Search for all licensed customers on GitLab.com we were thinking how to simplify the rollout of some Advanced Search features that require changing the data in Elasticsearch.

(If you're interested in the lessons we learned on our road to Enabling Elasticsearch for GitLab.com, you can read关于它的一切

数据迁移过程问题

Sometimes we need to change mappings of an index or backfill a field, and reindexing everything from scratch or usingZero downtime reindexing似乎是个显而易见的解决办法。但是,对于大型GitLab实例来说,这不是一个可伸缩的选项。GitLab.com是已知最大的GitLab安装,因此有很多项目、代码、问题、合并请求和其他需要索引的内容。例如,目前我们的Elasticsearch集群中有将近10亿个文档。重新索引所有内容需要数周甚至数月的时间,而且索引需要暂停,因此搜索结果很快就会过时。

多版本支持的原始计划

Originally, we were planning to introduce multi-version support using an approach that is fully reliant on GitLab to manage both indices, reading from the old one and writing to both until the migration is finished. You can read more information at!18254and&1769. 在撰写本文时,这种方法的大部分代码仍然以半实现的形式存在于GitLab中。

There were 2 primary concerns with this approach:

  1. Reindexing would require the GitLab application to read every single document from the storage and send it to Elasticsearch again. Doing so would put a big strain on different parts of the application, such as database, Gitaly, and Sidekiq.
  2. 在只需要更改索引的一小部分的情况下,重新索引从GitLab到集群的所有内容可能是非常浪费的。例如,如果我们想将epics添加到索引中,那么重新索引索引中的每个文档是非常浪费的,因为我们可以非常快速地索引所有epics。在许多情况下,我们将尝试执行一些迁移,这些迁移可以使用有针对性的方法更有效地完成(例如,向文档类型添加一个新字段只需要重新索引实际具有该字段的所有文档)。

基于这些原因,我们决定创建一个不同的数据迁移过程。

我们修订的数据迁移过程

We took inspiration from theRails DB migrations。We wanted to apply the best practices from it without having to re-architect what the Rails team has already implemented.

例如,我们决定使用一个带有时间戳的迁移文件的特殊目录。我们希望实现严格的执行顺序,以便可以同时运送许多迁移。一个特殊的后台处理人员将按计划检查此文件夹。这与rails后台迁移略有不同,后者要求操作员手动运行迁移。我们决定让它完全自动化并在后台运行,以避免自我管理的客户需要向迁移过程添加额外的步骤。这可能会让所有相关人员都更加困难,因为有很多方法可以运行GitLab。这个额外的限制还迫使我们总是认为迁移在代码中的任何一点都可能是不完整的,这对于零停机时间是至关重要的。

一开始,我们希望将迁移状态存储在Postgresql数据库中,但是我们决定不这样做,因为这对于用户希望将新的Elasticsearch集群连接到GitLab的情况来说可能并不完美。最好将迁移本身存储在Elasticsearch集群中,这样它们更有可能与数据同步。

You can see your new migration index in your Elasticsearch cluster. It's calledgitlab生manbetx体育客户端3.0产迁移。GitLab stores a few fields there. We use the version number as the document id. This is an example document:

{ "_id": "20210510143200", "_source": { "completed": true, "state": { }, "started_at": "2021-05-12T07:19:08.884Z", "completed_at": "2021-05-12T07:19:08.884Z" } }

The state field is used to store data that's required to run batched migrations. For example, for batched migrations we store a slice number and a task id for current Elasticsearch reindex operation and we update the state after every run.

迁移示例如下:

迁移名称<Elastic::Migration定义迁移# Migrate the data here结束定义完整的?# Return true if completed, otherwise return false结束结束

This looks a lot likeRails DB migrations从一开始这就是我们的目标。主要区别在于它有一个额外的方法来检查迁移是否完成。我们之所以添加这个方法,是因为我们需要经常执行异步任务,并且我们希望检查它是否稍后在另一个工作进程中完成。

Migration framework logic

This is a simple flow chart to demonstrate the high level logic of the new migration framework.

graph TD CRON(CRON every 30 minutes)-->| executes | WORKER[MigrationWorker]WORKER-->B(发现未完成的迁移)B-->HALT(已暂停)B-->UN(未完成)B-->COMP(已完成)HALT-->WARN(在管理UI中显示警告)WARN-->EX(exit)UN-->PREF(迁移预飞行检查)PREF-->RUN(执行迁移代码)COMP-->标记(标记为完工)标记-->EX

As you can see above, there are multiple different states of a migration. For example, the framework allows it to be halted when it has too many failed attempts. In that case, the warning will be shown in the admin UI with a button for restarting the migration.

How the warning looks like

Configuration options

We've introduced many useful configuration options into the framework, such as:

您可以在此开发中看到最新的选项列表documentation section

Data migration process results

We implemented the Advanced Search migration framework in the 13.6 release and have been improving it since. You can see some details in the original issue#234046。The only requirement for this new feature is that you should create your index using at least version 13.0. We have that requirement since we're heavily utilizing aliases, which were introduced in 13.0. As you might know, over the last few releases we've been working on separating different document types into their own indices. This migration framework has been a tremendous help for our initiative. We've already completed the migration of issues (in 13.8), comments (in 13.11), and merge requests (in 13.12) with a noticeable performance improvement.

Since we've accumulated so many different migrations over the last few releases and they require us to support multiple code paths for a long period of time, we've decided to remove older migrations that were added prior to the 13.12 release. You can see some details in thisissue。我们计划继续相同的策略在砰的一声ure, which is one of the reasons why you should always upgrade to the latest minor version before migrating to a major release.

如果您对需要高级搜索迁移的功能感兴趣,我们有专门的documentation sectionthat explains how to create one and lists all available options for it.

试用GitLab的所有功能-免费30天

GitLab is more than just source code management or CI/CD. It is a full software development lifecycle & DevOps tool in a single application.

1Manbetx
manbetx客户端打不开

Try2019新万博appmanbetⅩ risk-free for 30 days.

不需要信用卡。有问题吗?Contact us.

Gitlab x icon svg