
## So far so good

---

Web applications have always needed to run different tasks based on events triggered by the end-user. Long ago, when browsers were more restricted, these applications had to run tasks one by one in most cases.

Currently, browsers have more capabilities which allow more complex and demanding scenarios, such as the execution of concurrent tasks from a web application.

With the rise of **SPAs** (Single-Page Applications), some Javascript libraries or frameworks favor the use of [**Promises**](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise) while others favor [**Observables**](https://rxjs.dev/guide/observable). With either of them, it is possible to successfully manage the execution of tasks concurrently, either by using [**Promise.all**](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all) in the case of **Promises** or by using combination operators such as [**forkJoin**](https://rxjs.dev/api/index/function/forkJoin) in the case of **Observables**. All of these things make life easier, although it is not enough for some scenarios.

Let's imagine a SPA to store and tag videos where the end-user can upload videos while navigating through the different screens performing actions that request and load data. This specific case will be somewhat troublesome since all requests are [**XHR**](https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest) and, as is known, current browsers still have limitations in this regard.

If the end-user of this imaginary application needs to upload many videos in the background while still browsing it, we need some way to limit the number request running in the background to free up some connections to continue browsing, otherwise requests will start to be blocked once the limit set by the browser is exceeded. If you want to learn more about browser restrictions, click [here](https://www.linkedin.com/pulse/why-does-your-browser-limit-number-concurrent-ishwar-rimal/).

**RxJS** to the rescue! 🦸

## How the hell do we do this?

---

### Context

Since we are talking about uploading videos, we are going to represent the upload process by means of an entity called **process** that has a unique `id`, a `status` (`pending`, `in progress`, `completed` and `failed`), a `percentage` of progress (from `0` to `100`) and the metadata and content of the video in a `data` property.

```json
{
  "id": 1,
  "status": "pending",
  "percentage": 0,
  "data": {...}
}
```

### Concepts

Before we get down to business, it's important to know what a [**Subject**](https://rxjs.dev/guide/subject) is in the RxJS world.

In a nutshell, a **Subject** is a special type of **Observable** that allows multicasting unlike plain Observables that allow unicasting, but it also acts as an **Observer** allowing values to be sent through the **Subject** itself.

This piece of **RxJS** guides the rest of the lines in this post.

### Code

Let's to discover the code step by step for a better understanding.

First, we need a **Subject** that acts as the source where we are going to feed the new processes that have to be executed. For this purpose there is `addProcess$` which receives raw processes represented by the `id` and `data` properties mentioned above.

```js
const addProcess$ = new Subject();
```

Every time a new raw proccess is fed into the Subject, it goes trough a number of different status depending on the state of the concurrent queue that we are trying to build:

1. A new process comes in: the process remains with `status` equal `pending` and `percentage` equal to `0`.
2. A pending process is scheduled for execution: the process changes the `status` to `in progress` and `percentage` equal to the progress of the task, in this case, the HTTP request.
3. A running process completes or fails: the process changes the `status` to `completed` or `failed` depending on how it ends.

Keeping this in mind, we need a way to map the raw model to a meaninful model that allow us to represent that world.

```js
const addProcess$ = new Subject();
const processQueue$ = addProcess$
  .pipe(
    connect(subject$ =>
      subject$.pipe(
        mergeMap((process) => {
          return of(updateProcess(process, 'pending', 0));
        })
      )
    )
  ).subscribe(process => { ... })

const updateProcess = (process, status, percentage) => ({
  ...process,
  status,
  percentage,
});
```

The **connect** operator allows multicasting of the source, so it shares the single subscription created with other subscribers, which is efficient in terms of subscription management and, on the other hand, all the subscribers that may exist would receive the same data.

It's also necessary the use of **mergeMap** operator to project the value of the source to end up mapping to a model that represents the state mentioned above in the first point.

&nbsp;&nbsp;&nbsp;&nbsp;[x] A new process comes in: the process remains with `status` equal `pending` and `percentage` equal to `0`.

&nbsp;&nbsp;&nbsp;&nbsp;[ ] A pending process is scheduled for execution: the process changes the `status` to `in progress` and `percentage` equal to the progress of the task, in this case, the HTTP request.

&nbsp;&nbsp;&nbsp;&nbsp;[ ] A running process completes or fails: the process changes the `status` to `completed` or `failed` depending on how it ends.

To address the problem of scheduling the execution processes, let's take a look at the following code snippet 🤓:

```js
const maxConcurrency = 2;
const addProcess$ = new Subject();
const processQueue$ = addProcess$
  .pipe(
    connect(subject$ =>
      merge(
        subject$.pipe(
          mergeMap((process) => {
            return of(updateProcess(process, 'pending', 0))
              .pipe(
                observeOn(asyncScheduler),
                takeUntil(merge(subject$))
              );
          })
        ),
        subject$.pipe(
          mergeMap((process) => {
            return fakeHttpRequest(process)
              .pipe(
                startWith(updateProcess(process, 'in progress', 0)),
                endWith(updateProcess(process, 'completed', 100)),
                catchError((error) => of(updateProcess(process, 'failed', process.percentage))),
                scan((accum, value) => ({ ...accum, ...value }), {}),
              )
          }, maxConcurrency)
        )
      ).pipe(
        scan((accum, value) => ({ ...accum, [value.id]: value }), {}),
        map(obj => Object.values(obj))
      )
    )
  ).subscribe(processes => { ... })

const updateProcess = (process, status, percentage) => ({
  ...process,
  status,
  percentage,
});

const fakeHttpDelay = 5000;
const fakeHttpRequest = (process) => new Observable(subscriber => {
  subscriber.next(updateProcess(process, 'in progress', 25));
  setTimeout(() => {
    subscriber.next(updateProcess(process, 'in progress', 50));

    if (Math.floor(Math.random() * 10) % 2 === 0)
      subscriber.error(new Error('Connection error!'));

    setTimeout(() => {
      subscriber.next(updateProcess(process, 'in progress', 75));
      setTimeout(() => {
        subscriber.next(updateProcess(process, 'in progress', 99));
        subscriber.complete();
      }, fakeHttpDelay);
    }, fakeHttpDelay);
  }, fakeHttpDelay);
});
```

The first new thing we see is that just below the **connect** operator, there is a **merge** operator that allows us to declare another **Observable** to manage the scheduling of process executions.

The second **Observable** also uses a **mergeMap** operator to project the value of the source and subsequently fires an HTTP request (*fakeHttpRequest*) that simulates the process of uploading a video. Note that **mergeMap** receives the `maxConcurrency` variable as its second parameter, which establishes the maximum number of input **Observables** being subscribed to concurrently. This value sets how many upload processes are executed concurrently, leaving free connections to continue browsing.

As for the operators within the `pipe` block:

* **startWith** sets the state at the moment of subscription.
* **endWith** sets the state immediately after the source completes.
* **catchError** sets the state when an error occurs.
* **scan** allows managing the state by acummulating or merging intermediate states.

Before finishing, it's worth mentioning the pair of statements added to the first **Observable** inside the operator **merge**:
* **observeOn(asyncScheduler)** defer the emission of the data from the **Observable** to the JS event loop for better performance.
* **takeUntil(merge(subject$))** unsubscribe the **Observable** once it emits a value. The internal **merge** is a trick to make it emit once, since `subject$` has already emitted a value when it got here.

The above describes the states previously mentioned in the second and third points.

&nbsp;&nbsp;&nbsp;&nbsp;[x] A new process comes in: the process remains with `status` equal `pending` and `percentage` equal to `0`.

&nbsp;&nbsp;&nbsp;&nbsp;[x] A pending process is scheduled for execution: the process changes the `status` to `in progress` and `percentage` equal to the progress of the task, in this case, the HTTP request.

&nbsp;&nbsp;&nbsp;&nbsp;[x] A running process completes or fails: the process changes the `status` to `completed` or `failed` depending on how it ends.

### Bonus code

What if you want to drop some scheduled process on the queue? There goes a solution based on introducing a new Subject where the process to be discarded is notified. ✨

In a few words, merging what we already had with that new Subject and using a scan operator, we manage to discard those processes that are in `processQueue$` when they come through `dropProcess$`.

```js
const maxConcurrency = 2;
const addProcess$ = new Subject();
const dropProcess$ = new Subject();
const processQueue$ = addProcess$
  .pipe(
    connect(subject$ =>
      merge(
        subject$.pipe(
          mergeMap((process) => {
            return of(updateProcess(process, 'pending', 0))
              .pipe(
                observeOn(asyncScheduler),
                takeUntil(merge(subject$))
              );
          })
        ),
        subject$.pipe(
          mergeMap((process) => {
            return fakeHttpRequest(process)
              .pipe(
                startWith(updateProcess(process, 'in progress', 0)),
                endWith(updateProcess(process, 'completed', 100)),
                catchError((error) => of(updateProcess(process, 'failed', process.percentage))),
                scan((accum, value) => ({ ...accum, ...value }), {}),
                takeUntil(
                  dropProcess$.pipe(
                    filter(p => p.id === process.id)
                  )),
              )
          }, maxConcurrency)
        )
      )
    )
  );

const droppableProcessQueue$ = merge(processQueue$, dropProcess$)
  .pipe(
    scan((accum, value) => {
      if (value.status === 'dropped') {
        const { [value.id]: _, ...rest } = accum;
        return rest;
      }
      return { ...accum, [value.id]: value };
    }, {}),
    map(obj => Object.values(obj))
  );

droppableProcessQueue$.subscribe(processes => {...});

const updateProcess = (process, status, percentage) => ({
  ...process,
  status,
  percentage,
});

const fakeHttpDelay = 5000;
const fakeHttpRequest = (process) => new Observable(subscriber => {
  subscriber.next(updateProcess(process, 'in progress', 25));
  setTimeout(() => {
    subscriber.next(updateProcess(process, 'in progress', 50));

    if (Math.floor(Math.random() * 10) % 2 === 0)
      subscriber.error(new Error('Connection error!'));

    setTimeout(() => {
      subscriber.next(updateProcess(process, 'in progress', 75));
      setTimeout(() => {
        subscriber.next(updateProcess(process, 'in progress', 99));
        subscriber.complete();
      }, fakeHttpDelay);
    }, fakeHttpDelay);
  }, fakeHttpDelay);
});
```

### Sample application

Below is a screenshot of an application that implements all the code described in the post for testing purposes.

![app](/images/concurrent-task-queue-using-rxjs/sample-app.jpg)

The full source code of the sample application is [**here**](https://github.com/jcarloslr10/concurrent-task-queue-using-rxjs).

## Conclusions

---

To wrap this up:

- The power of the RxJS library and all its components and operators is evident, it really provides a range of fascinating options.

- It becomes clear that in many scenarios RxJS is as powerful as it's complex. Sometimes, a deep knowledge is needed when implementing this type of solutions.

I hope you like it and it clarifies a lot of the black magic that RxJS has.

See you! 😉


Implementation of a queue to execute asynchronous tasks tuning the degree of parallelism.

Concurrent task queue using RxJS


> Thanks to my teammate [*Daniel Perdomo*](https://www.linkedin.com/in/daniel-perdomo-lorenzo-35b5a070/) for his help. I couldn't have done it without you. 🙏

## The Millenium Problem

---

Some of us have worked building applications where, at some point, they need to filter data based on a specified pattern. At first glance, it may seem like a simple problem to solve, but it can hide a poison dart behind it.

It all starts when a stakeholder tells us that the end-user has a need to filter data based on text matching. We, as conscientious software engineer, ask the following questions:

> "Does it have to be an exact text matching?"

> "Does it have to be a **smarter** text matching by taking singular and plural terms into account and skipping stop words?"

If the answer to the **second question is yes**, we should consider paths related to [**text indexes**](https://www.mongodb.com/docs/manual/core/index-text/) in MongoDB or, for more demanding scenarios, other technologies such as [**Atlas Search**](https://www.mongodb.com/docs/atlas/atlas-search/) or [**Elasticsearch**](https://www.elastic.co/what-is/elasticsearch).

If the answer to the **first question is yes** then we should consider use [**regular indexes**](https://www.mongodb.com/docs/manual/indexes/), although we still have one more question to ask:

> "Can the pattern match only against the values from the beginning of the string (**prefix pattern**) or anywhere (**LIKE-style**)?"

Fortunately, if the answer to the question is *"It only matches against the values from the beginning..."* then we just dodged a bullet 😁 because filtering prefix patterns over text index in MongoDB is the least bad thing that can happen to us since it is efficient enough. But, if the answer to the question is *"Anywhere..."*, then we are in trouble. 😥

By the way, before continuing, the stakeholders also ask us to make the matchings insensitive to uppercase, lowercase and diacritics... 😭

## Regex operator

---

First, I would like to introduce the `$regex` operator in MongoDB. The operator accepts a regular expression of the following form `/pattern/<options>`. An example of non-prefix pattern might be `/cor/i` which looks for the term **cor** anywhere in the string and insensitively.

This type of approach poses two problems, even using regular indexes:

1. The use of non-prefix pattern does not allow setting boundaries when going through the data to be filtered.
2. The option flag `i` causes MongoDB to perform insensitive comparisons which is more inefficient.

## Use case

---

The application that we are building has a list of users. A end-user of this application has to be able to search users by full name, but to perform meaningful searches on the data, at least one term of **at least three alphanumeric characters** must be typed into the text search box.

This limitation on the minimum size of the term is set because searches for **below three characters** will not yield consistent and useful results.

Examples of **correct** searches:
`Bry`
`Bryan`
`cor Be`

Examples of **incorrect** searches:
`b`
`Br`
`r b`
`or Be`

#### User model

The below JSON represents a user in the `users` collection in MongoDB.

```json
{
  "_id": { "$oid":"626d70bcc30933f61a0786e3"},
  "name":"Corey Batz",
  "job":"Legacy Usability Consultant"
}
```

#### User query

The below Javascript represents how the data are queried using an insensitive non-prefix pattern.

```js
db.getCollection("users").aggregate([
    { $match: { name: { $regex: /corey ba/i } } }
])
```

Next, we are going to run the queries on the `users` collection which has **1,000,000 documents**, so stay tuned.

## Avoiding insensitive flag

---

Building on the user data model above, a new field called `normalizedName` could be added to store the normalized name (lowercase, no diacritics, single space) on which to match later. That may be possible by normalizing the search terms before trying to query them.

So we would have the below JSON representation in the `users` collection:

```json
{
  "_id": { "$oid":"626d70bcc30933f61a0786e3"},
  "name":"Corey Batz",
  "normalizedName":"corey batz",
  "job":"Legacy Usability Consultant"
}
```

And the query would look like this:

```js
db.getCollection("users").aggregate([
    { $match: { normalizedName: { $regex: /corey ba/ } } }
])
```

At least this solution will prevent MongoDB from having to perform insensitive comparisons, although it is not enough to guarantee proper operation in a production environment.

It seems we will have to create an index to make the `$regex` expression more efficient because so far MongoDB is scanning the collection. Let's go there!

```js
db.getCollection("users").createIndex(
  { normalizedName: 1 },
  { name: "ix_normalizedName" }
);
```

If we run the query again and take a look at the query plan we can see the following:

```json
{
  "explainVersion": "1",
  "queryPlanner": {
    "namespace": "crm.users",
    "indexFilterSet": false,
    "parsedQuery": {
      "normalizedName": {
        "$regex": "rey li"
      }
    },
    "optimizedPipeline": true,
    "maxIndexedOrSolutionsReached": false,
    "maxIndexedAndSolutionsReached": false,
    "maxScansToExplodeReached": false,
    "winningPlan": {
      "stage": "FETCH",
      "inputStage": {
        "stage": "IXSCAN",
        "filter": {
          "normalizedName": {
            "$regex": "rey li"
          }
        },
        "keyPattern": {
          "normalizedName": 1.0
        },
        "indexName": "ix_normalizedName",
        "isMultiKey": false,
        "multiKeyPaths": {
          "normalizedName": []
        },
        "isUnique": false,
        "isSparse": false,
        "isPartial": false,
        "indexVersion": 2.0,
        "direction": "forward",
        "indexBounds": {
          "normalizedName": [
            "[\"\", {})",
            "[/rey li/, /rey li/]"
          ]
        }
      }
    },
    "rejectedPlans": []
  },
  "executionStats": {
    "executionSuccess": true,
    "nReturned": 40.0,
    "executionTimeMillis": 1048.0,
    "totalKeysExamined": 1000000.0,
    "totalDocsExamined": 40.0,
    "executionStages": {
      "stage": "FETCH",
      "nReturned": 40.0,
      "executionTimeMillisEstimate": 93.0,
      "works": 1000001.0,
      "advanced": 40.0,
      "needTime": 999960.0,
      "needYield": 0.0,
      "saveState": 1000.0,
      "restoreState": 1000.0,
      "isEOF": 1.0,
      "docsExamined": 40.0,
      "alreadyHasObj": 0.0,
      "inputStage": {
        "stage": "IXSCAN",
        "filter": {
          "normalizedName": {
            "$regex": "rey li"
          }
        },
        "nReturned": 40.0,
        "executionTimeMillisEstimate": 92.0,
        "works": 1000001.0,
        "advanced": 40.0,
        "needTime": 999960.0,
        "needYield": 0.0,
        "saveState": 1000.0,
        "restoreState": 1000.0,
        "isEOF": 1.0,
        "keyPattern": {
          "normalizedName": 1.0
        },
        "indexName": "ix_normalizedName",
        "isMultiKey": false,
        "multiKeyPaths": {
          "normalizedName": []
        },
        "isUnique": false,
        "isSparse": false,
        "isPartial": false,
        "indexVersion": 2.0,
        "direction": "forward",
        "indexBounds": {
          "normalizedName": [
            "[\"\", {})",
            "[/rey li/, /rey li/]"
          ]
        },
        "keysExamined": 1000000.0,
        "seeks": 1.0,
        "dupsTested": 0.0,
        "dupsDropped": 0.0
      }
    },
    "allPlansExecution": []
  },
  "command": {
    "aggregate": "users",
    "pipeline": [
      {
        "$match": {
          "normalizedName": {
            "$regex": /rey li/
          }
        }
      }
    ],
    "cursor": {},
    "$db": "crm"
  },
  "serverInfo": {
    ...
  },
  "serverParameters": {
    ...
  },
  "ok": 1.0
}
```

Analyzing the query plan we see that MongoDB takes the previously created index as the first stage of the winning plan:

`queryPlanner.winningPlan.inputStage.stage: "IXSCAN"`
`queryPlanner.winningPlan.inputStage.indexName: "ix_normalizedName"`

That's good! But what about the amount of work using the index?

Well, if we see the total number of index key examined, we will be stunned. MongoDB scanned the whole index (**1,000,000 keys**) to end up returning **40 documents** and it took 1,048 milliseconds to execute it. 🥴

`executionStats.executionTimeMillis: 1048.0`

`executionStats.totalKeysExamined: 1000000.0`

`executionStats.totalDocsExamined: 40.0`

`executionStats.nReturned: 40.0`

As the `$regex` is non-prefix expression, MongoDB cannot optimize the `IXSCAN` stage setting bounds when scanning the index, so we need a strategy for MongoDB to be able to set these bounds to do the index scan and return results faster.

## Querying efficient non-prefix pattern

---

As stated before, it's assumed that at least three alphanumeric character term is typed into text search box in order to perform meaningful searches, so let's define the strategy that will change our lives. 🤨

### The Algorithm 🚀

Since we only perform searches if the search term has three alphanumeric characters, we could add a new field called `normalizedNameChunks` to store all the three-character chunks of the `normalizedName` field including words with less than three characters as well. Let me explain:

Taking the `normalizedName` **"corey li batz"** as an example, we would do the following:

1. Split the `normalizedName` by spaces, outputting: `["corey", "li", "batz"]`
2. Divide each word into as many three-character chunks as possible and keep words less than three characters as is, outputting: `["cor", "ore", "rey", "li", "bat", "atz"]`

```json
{
  "_id": { "$oid":"626d70bcc30933f61a0786e3"},
  "name":"Corey Li Batz",
  "normalizedName":"corey li batz",
  "normalizedNameChunks":["cor", "ore", "rey", "li", "bat", "atz"],
  "job":"Legacy Usability Consultant"
}
```

Using this approach, we could search for matches on this new field, but first we would need to apply the same steps to the search text input.

Taking the search text input **"rey li"** as an example, we would do the following:

1. Split the text input by spaces, outputting: `["rey", "li"]`
2. Divide each word into as many three-character chunks as possible and keep words less than three characters as is, outputting: `["rey", "li"]`

Thus, we could build a query that finds an intersection between the array of the `normalizedNameChunks` field and the array of the text input.

`["cor", "ore", "rey", "li", "bat", "atz"] ∩ ["rey", "li"] = ["rey"]`

Using the `$in` array operator we manage to perform the intersection operation between array field and input array, creating the query as follows:

```js
db.getCollection("users").aggregate([
    { $match: { normalizedNameChunks: { $in: ["rey", "li"] } } },
])
```

The intersection operations means that there are matches between the arrays, but the problem with this new query is that it does not guarantee that the order of the chunks is correct, so we need to do something else on the query:

```js
db.getCollection("users").aggregate([
    { $match: { normalizedNameChunks: { $in: ["rey", "li"] } } },
    { $match: { normalizedName: { $regex: /rey li/ } } }
])
```

The new stage filtering by `normalizedName` field using `$regex` expression will ensure that the order of the chunks is correct.

One more time as we have done before, we need to create a new index to support the new query conditions on `normalizedNameChunks` and `normalizedName` fields:

```js
db.getCollection("users").createIndex(
  { normalizedNameChunks: 1, normalizedName: 1 },
  { name: "ix_normalizedNameChunks_normalizedName" }
);
```

Now we really have it all! Let's see the query plan again.

```json
{
  "explainVersion": "1",
  "queryPlanner": {
    "namespace": "crm.users",
    "indexFilterSet": false,
    "parsedQuery": {
      "$and": [
        {
          "normalizedName": {
            "$regex": "rey li"
          }
        },
        {
          "normalizedNameChunks": {
            "$in": [
              "li",
              "rey"
            ]
          }
        }
      ]
    },
    "optimizedPipeline": true,
    "maxIndexedOrSolutionsReached": false,
    "maxIndexedAndSolutionsReached": false,
    "maxScansToExplodeReached": false,
    "winningPlan": {
      "stage": "FETCH",
      "filter": {
        "normalizedName": {
          "$regex": "rey li"
        }
      },
      "inputStage": {
        "stage": "IXSCAN",
        "keyPattern": {
          "normalizedNameChunks": 1.0,
          "normalizedName": 1.0
        },
        "indexName": "ix_normalizedNameChunks_normalizedName",
        "isMultiKey": true,
        "multiKeyPaths": {
          "normalizedNameChunks": [
            "normalizedNameChunks"
          ],
          "normalizedName": []
        },
        "isUnique": false,
        "isSparse": false,
        "isPartial": false,
        "indexVersion": 2.0,
        "direction": "forward",
        "indexBounds": {
          "normalizedNameChunks": [
            "[\"li\", \"li\"]",
            "[\"rey\", \"rey\"]"
          ],
          "normalizedName": [
            "[\"\", {})",
            "[/rey li/, /rey li/]"
          ]
        }
      }
    },
    "rejectedPlans": []
  },
  "executionStats": {
    "executionSuccess": true,
    "nReturned": 40.0,
    "executionTimeMillis": 278.0,
    "totalKeysExamined": 7018.0,
    "totalDocsExamined": 7016.0,
    "executionStages": {
      "stage": "FETCH",
      "filter": {
        "normalizedName": {
          "$regex": "rey li"
        }
      },
      "nReturned": 40.0,
      "executionTimeMillisEstimate": 207.0,
      "works": 7019.0,
      "advanced": 40.0,
      "needTime": 6977.0,
      "needYield": 0.0,
      "saveState": 17.0,
      "restoreState": 17.0,
      "isEOF": 1.0,
      "docsExamined": 7016.0,
      "alreadyHasObj": 0.0,
      "inputStage": {
        "stage": "IXSCAN",
        "nReturned": 7016.0,
        "executionTimeMillisEstimate": 10.0,
        "works": 7018.0,
        "advanced": 7016.0,
        "needTime": 1.0,
        "needYield": 0.0,
        "saveState": 17.0,
        "restoreState": 17.0,
        "isEOF": 1.0,
        "keyPattern": {
          "normalizedNameChunks": 1.0,
          "normalizedName": 1.0
        },
        "indexName": "ix_normalizedNameChunks_normalizedName",
        "isMultiKey": true,
        "multiKeyPaths": {
          "normalizedNameChunks": [
            "normalizedNameChunks"
          ],
          "normalizedName": []
        },
        "isUnique": false,
        "isSparse": false,
        "isPartial": false,
        "indexVersion": 2.0,
        "direction": "forward",
        "indexBounds": {
          "normalizedNameChunks": [
            "[\"li\", \"li\"]",
            "[\"rey\", \"rey\"]"
          ],
          "normalizedName": [
            "[\"\", {})",
            "[/rey li/, /rey li/]"
          ]
        },
        "keysExamined": 7018.0,
        "seeks": 2.0,
        "dupsTested": 7016.0,
        "dupsDropped": 0.0
      }
    },
    "allPlansExecution": []
  },
  "command": {
    "aggregate": "users",
    "pipeline": [
      {
        "$match": {
          "normalizedNameChunks": {
            "$in": [
              "rey",
              "li"
            ]
          }
        }
      },
      {
        "$match": {
          "normalizedName": {
            "$regex": /rey li/
          }
        }
      }
    ],
    "cursor": {},
    "$db": "crm"
  },
  "serverInfo": {
    ...
  },
  "serverParameters": {
    ...
  },
  "ok": 1.0
}
```

Analyzing the query plan we see that MongoDB takes the previously created index as the first stage of the winning plan:

`queryPlanner.winningPlan.inputStage.stage: "IXSCAN"`
`queryPlanner.winningPlan.inputStage.indexName: "ix_normalizedNameChunks_normalizedName"`

That's awesome! But what about the amount of work using this index?

Well, if we see the total number of index key examined, we will be amazed. MongoDB scanned the index more efficiently setting bounds (**7,018 keys**) to end up returning **40 documents** and it took 278 milliseconds to execute it. Nice! 😎

`executionStats.executionTimeMillis: 278.0`

`executionStats.totalKeysExamined: 7018.0`

`executionStats.totalDocsExamined: 7016.0`

`executionStats.nReturned: 40.0`

## Conclusions

---

In summary, it is important to keep in mind the following key points:

- Avoid using inefficient regular expression patterns like non-prefix patterns `/batz/` or insensitive patterns `/batz/i`.

- Try to use prefix patterns like `/^corey/` or `/batz$/` when using regular expressions so MongoDB can set bounds for scanning index.

- Try to use a different field to store the normalized value of a given field in order to perform a insensitive comparison in terms of casing and diacritics.

- Splitting words into chunks of a defined minimum length allow us to perform much more efficient first stage.

- An *inefficient* second stage on a much smaller data set is much more *efficient* than an *inefficient* first stage on a very large data set.



How to leash the regex beast using a simple approach.