Introduction to Elasticsearch: Full-Text Search Engine

Introduction

Elasticsearch is a distributed, RESTful search and analytics engine built on Apache Lucene. Since its release in 2010 by Shay Banon, it has become the most widely deployed search engine in the world, powering search functionality at companies like Wikipedia, GitHub, Stack Overflow, and Netflix. Elasticsearch excels at full-text search, log analytics, real-time application monitoring, and security analytics—any use case that requires fast, relevant results from large volumes of structured or unstructured data.

Unlike traditional relational databases that use B-tree indexes optimized for exact matches, Elasticsearch uses an inverted index structure optimized for text search. When you index a document, Elasticsearch analyzes the text—tokenizing it into words, normalizing case, removing stop words, and applying stemming—then stores each unique term with pointers to every document containing it. This makes search queries extraordinarily fast, even across millions of documents.

Elasticsearch is part of the Elastic Stack (formerly ELK Stack), which includes Logstash for data ingestion, Kibana for visualization, and Beats for lightweight data shippers. Together, these tools provide a complete solution for searching, analyzing, and visualizing data in real time. This guide covers Elasticsearch's architecture, practical setup, query DSL, aggregations, and integration with Node.js applications.

Search engine and data indexing architecture

Understanding Elasticsearch: Core Architecture

The Inverted Index

The inverted index is the foundation of Elasticsearch's search capability. Unlike a forward index that maps documents to their terms (Document → Terms), an inverted index maps terms to documents (Term → Documents). When you search for "typescript tutorial," Elasticsearch looks up both terms in the inverted index, finds documents containing them, and ranks results by relevance.

The analysis pipeline processes text before indexing: a character filter removes HTML tags or special characters, a tokenizer splits text into terms (by whitespace, by word boundaries, or with a custom pattern), and token filters transform terms (lowercasing, stemming, synonym expansion). This pipeline is configurable per field, enabling different analysis strategies for different content types.

Clusters, Nodes, and Shards

An Elasticsearch cluster consists of one or more nodes (servers). Each node plays one or more roles: master node (manages cluster state), data node (stores data and executes queries), ingest node (pre-processes documents), or coordinating node (routes requests). For production, dedicated master and data nodes are recommended.

Data is distributed across shards—each index is split into a configurable number of primary shards. Each primary shard can have zero or more replica shards for redundancy and read scaling. A document lives in exactly one primary shard (determined by hashing the document ID) and its replicas. When a node fails, replicas on other nodes are promoted to primaries automatically.

Mapping: Schema Definition

Elasticsearch uses mappings to define how documents and their fields are stored and indexed. Field types include text (analyzed full-text), keyword (exact value, not analyzed), integer, float, date, boolean, nested (for arrays of objects), and geo_point (for location data). Dynamic mapping automatically infers types from data, but explicit mapping is recommended for production.

// Explicit mapping for a product index
{
  "mappings": {
    "properties": {
      "name": { "type": "text", "analyzer": "standard", "fields": { "keyword": { "type": "keyword" } } },
      "description": { "type": "text", "analyzer": "english" },
      "price": { "type": "float" },
      "category": { "type": "keyword" },
      "tags": { "type": "keyword" },
      "created_at": { "type": "date" },
      "location": { "type": "geo_point" },
      "reviews": {
        "type": "nested",
        "properties": {
          "author": { "type": "keyword" },
          "rating": { "type": "integer" },
          "comment": { "type": "text" }
        }
      }
    }
  }
}

Architecture and Design Patterns

The Search Pattern

Elasticsearch queries follow a two-phase execution: query phase (each shard returns document IDs and relevance scores) and fetch phase (the coordinating node retrieves full documents from relevant shards). This distributed architecture enables horizontal scaling—adding nodes increases both storage capacity and query throughput.

The Aggregation Pattern

Aggregations provide real-time analytics over your data: bucket aggregations group documents by field values (like SQL GROUP BY), metric aggregations compute statistics (sum, avg, min, max, percentiles), and pipeline aggregations compute over other aggregations (like moving averages). Aggregations execute in a single pass alongside search queries, enabling faceted search and real-time dashboards.

The Index Lifecycle Pattern

Index Lifecycle Management (ILM) automates index management: hot indices on fast storage for recent data, warm indices on slower storage for older data, and cold/frozen indices for archival. ILM policies automatically roll over indices by size or age, reduce replicas, force-merge segments, and delete expired data.

Step-by-Step Implementation

Let's build a complete Elasticsearch application with Node.js.

Setting Up Elasticsearch with Docker

# docker-compose.yml
version: '3.8'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ports:
      - "9200:9200"
    volumes:
      - es-data:/usr/share/elasticsearch/data
 
  kibana:
    image: docker.elastic.co/kibana/kibana:8.12.0
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch
 
volumes:
  es-data:

Connecting with the Node.js Client

import { Client } from "@elastic/elasticsearch";
 
const client = new Client({
  node: "http://localhost:9200",
  // For production with authentication:
  // auth: { username: "elastic", password: "changeme" },
  // tls: { rejectUnauthorized: false }
});

Creating an Index with Mapping

async function createProductIndex() {
  const exists = await client.indices.exists({ index: "products" });
  if (exists) {
    console.log("Index already exists");
    return;
  }
 
  await client.indices.create({
    index: "products",
    settings: {
      number_of_shards: 2,
      number_of_replicas: 1,
      analysis: {
        analyzer: {
          product_analyzer: {
            type: "custom",
            tokenizer: "standard",
            filter: ["lowercase", "stop", "snowball"],
          },
        },
      },
    },
    mappings: {
      properties: {
        name: {
          type: "text",
          analyzer: "product_analyzer",
          fields: { keyword: { type: "keyword" } },
        },
        description: { type: "text", analyzer: "product_analyzer" },
        price: { type: "float" },
        category: { type: "keyword" },
        brand: { type: "keyword" },
        tags: { type: "keyword" },
        in_stock: { type: "boolean" },
        created_at: { type: "date" },
        rating: { type: "float" },
        reviews_count: { type: "integer" },
      },
    },
  });
 
  console.log("Product index created");
}

Indexing Documents

interface Product {
  name: string;
  description: string;
  price: number;
  category: string;
  brand: string;
  tags: string[];
  in_stock: boolean;
  rating: number;
  reviews_count: number;
}
 
async function indexProduct(id: string, product: Product) {
  await client.index({
    index: "products",
    id,
    document: {
      ...product,
      created_at: new Date().toISOString(),
    },
  });
}
 
// Bulk indexing for high throughput
async function bulkIndexProducts(products: Array<{ id: string; product: Product }>) {
  const operations = products.flatMap(({ id, product }) => [
    { index: { _index: "products", _id: id } },
    { ...product, created_at: new Date().toISOString() },
  ]);
 
  const result = await client.bulk({ operations, refresh: true });
 
  if (result.errors) {
    const errored = result.items.filter(item => item.index?.error);
    console.error(`Failed to index ${errored.length} documents`);
    errored.forEach(item => console.error(item.index?.error));
  } else {
    console.log(`Successfully indexed ${products.length} products`);
  }
}

Searching with Query DSL

async function searchProducts(query: string, filters: {
  category?: string;
  minPrice?: number;
  maxPrice?: number;
  inStock?: boolean;
  page?: number;
  size?: number;
} = {}) {
  const { category, minPrice, maxPrice, inStock, page = 0, size = 20 } = filters;
 
  const result = await client.search({
    index: "products",
    body: {
      from: page * size,
      size,
      query: {
        bool: {
          must: [
            {
              multi_match: {
                query,
                fields: ["name^3", "description", "tags^2", "brand"],
                type: "best_fields",
                fuzziness: "AUTO",
              },
            },
          ],
          filter: [
            ...(category ? [{ term: { category } }] : []),
            ...(minPrice || maxPrice
              ? [{
                  range: {
                    price: {
                      ...(minPrice ? { gte: minPrice } : {}),
                      ...(maxPrice ? { lte: maxPrice } : {}),
                    },
                  },
                }]
              : []),
            ...(inStock !== undefined ? [{ term: { in_stock: inStock } }] : []),
          ],
        },
      },
      sort: [
        { _score: "desc" },
        { rating: "desc" },
      ],
      highlight: {
        fields: {
          name: {},
          description: { fragment_size: 200 },
        },
        pre_tags: ["<mark>"],
        post_tags: ["</mark>"],
      },
      aggs: {
        categories: {
          terms: { field: "category", size: 20 },
        },
        price_ranges: {
          range: {
            field: "price",
            ranges: [
              { to: 25 },
              { from: 25, to: 50 },
              { from: 50, to: 100 },
              { from: 100 },
            ],
          },
        },
        avg_rating: {
          avg: { field: "rating" },
        },
        brands: {
          terms: { field: "brand", size: 10 },
        },
      },
    },
  });
 
  return {
    total: result.hits.total,
    products: result.hits.hits.map(hit => ({
      id: hit._id,
      score: hit._score,
      ...hit._source,
      highlights: hit.highlight,
    })),
    facets: {
      categories: result.aggregations?.categories,
      priceRanges: result.aggregations?.price_ranges,
      avgRating: result.aggregations?.avg_rating,
      brands: result.aggregations?.brands,
    },
  };
}

Building a REST API with Elasticsearch

import express from "express";
 
const app = express();
app.use(express.json());
 
app.get("/api/search", async (req, res) => {
  const { q, category, minPrice, maxPrice, inStock, page, size } = req.query;
 
  if (!q) {
    return res.status(400).json({ error: "Query parameter 'q' is required" });
  }
 
  try {
    const results = await searchProducts(q as string, {
      category: category as string,
      minPrice: minPrice ? parseFloat(minPrice as string) : undefined,
      maxPrice: maxPrice ? parseFloat(maxPrice as string) : undefined,
      inStock: inStock ? inStock === "true" : undefined,
      page: page ? parseInt(page as string) : 0,
      size: size ? parseInt(size as string) : 20,
    });
 
    res.json(results);
  } catch (error) {
    console.error("Search error:", error);
    res.status(500).json({ error: "Search failed" });
  }
});
 
app.post("/api/products", async (req, res) => {
  try {
    const id = crypto.randomUUID();
    await indexProduct(id, req.body);
    res.status(201).json({ id, ...req.body });
  } catch (error) {
    console.error("Index error:", error);
    res.status(500).json({ error: "Failed to index product" });
  }
});
 
app.listen(3000, () => console.log("API running on port 3000"));

Real-World Use Cases

Use Case 1: E-Commerce Product Search

Elasticsearch powers product search for major e-commerce platforms. Features include autocomplete suggestions (using completion suggesters), faceted search (filtering by category, price, brand), typo tolerance (fuzzy matching), and personalized ranking based on user behavior. Amazon's search uses similar inverted index technology.

Use Case 2: Log Analytics with the ELK Stack

Organizations ship application logs, server metrics, and security events to Elasticsearch via Logstash or Beats. Kibana dashboards provide real-time visualization of error rates, response times, and user behavior. This setup enables rapid incident response—engineers can search across millions of log entries in milliseconds.

Use Case 3: Application Performance Monitoring (APM)

Elastic APM traces requests across microservices, capturing latency, errors, and resource usage. This enables developers to identify performance bottlenecks, trace errors to their root cause, and monitor service health—all within the Elasticsearch ecosystem.

Use Case 4: Enterprise Search

Organizations use Elasticsearch to build internal search platforms that index documents from SharePoint, Confluence, Google Drive, and custom databases. The search engine provides unified results across all data sources with relevance ranking and access control.

Best Practices for Production

Define explicit mappings: Don't rely on dynamic mapping in production. Define field types, analyzers, and index settings explicitly. Dynamic mapping can create incorrect field types (e.g., treating numeric strings as integers).
Use bulk API for batch operations: Individual index requests create one HTTP roundtrip per document. The bulk API batches up to 1000 operations per request, dramatically improving throughput.
Right-size shards: Aim for shard sizes between 10GB and 50GB. Too many small shards increase overhead; too few large shards limit parallelism and recovery time. One shard per node is a good starting point.
Use refresh intervals wisely: By default, Elasticsearch refreshes indices every second, making new documents searchable. For bulk indexing operations, set refresh_interval: -1 during indexing and restore it afterward for better performance.
Implement circuit breakers: Elasticsearch has built-in circuit breakers to prevent out-of-memory errors. Monitor heap usage and tune breaker limits based on your workload.
Use aliases for zero-downtime reindexing: Create indices with versioned names (e.g., products-v2) and point aliases (e.g., products) at them. Reindex to a new version and swap the alias for zero-downtime schema changes.
Monitor cluster health: Use the _cluster/health API and Kibana's monitoring dashboards to track cluster status (green/yellow/red), shard allocation, and node resources.
Secure with authentication and encryption: Enable X-Pack security for authentication, role-based access control, and TLS encryption in production. Never expose Elasticsearch directly to the internet.

Common Pitfalls and Solutions

Pitfall	Impact	Solution
Too many shards per index	High overhead, slow recovery	Start with 1-2 shards per index; scale with replicas
Searching on `text` fields for exact matches	Unexpected results	Use `.keyword` sub-field for exact matches
Not using `filter` context	Unnecessary scoring overhead	Use `filter` for non-scoring queries (exact matches, ranges)
Ignoring mapping conflicts	Data type errors, failed queries	Define explicit mappings and validate before indexing
Storing large documents	Increased storage, slower queries	Store only searchable fields; keep full documents in a separate store

Performance Optimization

// Elasticsearch performance optimization
 
// 1. Use filter context for non-scoring queries (cached)
const optimizedQuery = {
  query: {
    bool: {
      must: { match: { description: "wireless headphones" } },
      filter: [
        { term: { category: "electronics" } },     // Cached, fast
        { range: { price: { gte: 50, lte: 200 } } }, // Cached, fast
        { term: { in_stock: true } },               // Cached, fast
      ],
    },
  },
};
 
// 2. Use source filtering to reduce payload
const result = await client.search({
  index: "products",
  body: {
    _source: ["name", "price", "category"], // Only return these fields
    query: { match_all: {} },
  },
});
 
// 3. Use scroll API for large result sets
const scrollSearch = client.helpers.scrollSearch({
  index: "products",
  body: { size: 1000, query: { match_all: {} } },
});
 
for await (const result of scrollSearch) {
  for (const hit of result.hits.hits) {
    await processDocument(hit._source);
  }
}
 
// 4. Optimize with pre-filtering
// Instead of complex queries with many clauses, use a filter cache
const cachedFilter = {
  bool: {
    filter: [
      { terms: { category: ["electronics", "computers"] } },
      { range: { price: { gte: 0, lte: 1000 } } },
    ],
  },
};

Comparison with Alternatives

Feature	Elasticsearch	Algolia	Typesense	PostgreSQL FTS
Hosting	Self-managed or Elastic Cloud	Managed only	Self-managed or managed	Self-managed
Full-text search	Excellent	Excellent	Excellent	Good
Typo tolerance	Fuzzy matching	Built-in	Built-in	Limited
Faceted search	Aggregations	Built-in filters	Facets	Manual
Real-time indexing	Near real-time (1s)	Real-time	Real-time	Real-time
Scaling	Horizontal (sharding)	Automatic	Manual sharding	Vertical
Cost	Infrastructure + team	Per-search pricing	Infrastructure	Included with PostgreSQL
Complexity	High	Low	Medium	Low
Analytics	Built-in	Limited	Limited	Manual

Elasticsearch excels when you need full control, complex queries, and analytics alongside search. Algolia provides the best out-of-box search experience with minimal setup. Typesense offers a simpler self-hosted alternative. PostgreSQL FTS is sufficient for simple search needs when you're already using PostgreSQL.

Advanced Patterns and Techniques

Autocomplete with Completion Suggester

// Index with completion field
await client.indices.create({
  index: "suggestions",
  mappings: {
    properties: {
      suggest: {
        type: "completion",
        analyzer: "simple",
        search_analyzer: "simple",
      },
      title: { type: "text" },
    },
  },
});
 
// Index suggestions
await client.index({
  index: "suggestions",
  document: {
    suggest: {
      input: ["typescript", "ts", "TypeScript programming"],
      weight: 10,
    },
    title: "TypeScript Programming Guide",
  },
});
 
// Autocomplete query
async function autocomplete(prefix: string) {
  const result = await client.search({
    index: "suggestions",
    body: {
      suggest: {
        autocomplete: {
          prefix,
          completion: {
            field: "suggest",
            fuzzy: { fuzziness: "AUTO" },
            size: 10,
          },
        },
      },
    },
  });
 
  return result.suggest.autocomplete[0].options.map(option => ({
    text: option.text,
    score: option._score,
  }));
}

Multi-Index Search

async function globalSearch(query: string) {
  const result = await client.search({
    index: ["products", "articles", "users"],
    body: {
      query: {
        multi_match: {
          query,
          fields: ["name^3", "title^3", "description", "content"],
          type: "best_fields",
          tie_breaker: 0.3,
        },
      },
      indices_boost: [
        { "products": 2 },
        { "articles": 1.5 },
        { "users": 1 },
      ],
    },
  });
 
  return result.hits.hits.map(hit => ({
    index: hit._index,
    id: hit._id,
    score: hit._score,
    ...hit._source,
  }));
}

Testing Strategies

import { Client } from "@elastic/elasticsearch";
 
const testClient = new Client({ node: "http://localhost:9200" });
 
describe("Elasticsearch Integration", () => {
  const testIndex = `test-products-${Date.now()}`;
 
  beforeAll(async () => {
    await testClient.indices.create({
      index: testIndex,
      body: {
        mappings: {
          properties: {
            name: { type: "text" },
            price: { type: "float" },
            category: { type: "keyword" },
          },
        },
      },
    });
  });
 
  afterAll(async () => {
    await testClient.indices.delete({ index: testIndex });
  });
 
  test("indexes and searches documents", async () => {
    await testClient.index({
      index: testIndex,
      id: "1",
      document: { name: "Wireless Headphones", price: 99.99, category: "electronics" },
      refresh: true,
    });
 
    const result = await testClient.search({
      index: testIndex,
      body: { query: { match: { name: "wireless" } } },
    });
 
    expect(result.hits.hits).toHaveLength(1);
    expect(result.hits.hits[0]._source.name).toBe("Wireless Headphones");
  });
});

Future Outlook

Elasticsearch continues to evolve with significant improvements in vector search (for AI/ML workloads), serverless deployment options, and enhanced security features. The introduction of Elastic Serverless makes it easier to deploy and scale Elasticsearch without managing infrastructure. Vector search capabilities enable semantic search using embeddings from language models, bridging traditional keyword search with AI-powered understanding.

ES|QL (Elasticsearch Query Language) provides a more intuitive query syntax for analytics, and the continued investment in Lucene improvements ensures Elasticsearch remains at the cutting edge of search technology.

Conclusion

Elasticsearch is a powerful, scalable search and analytics engine that enables fast, relevant search across large datasets. Its inverted index structure, distributed architecture, and rich query DSL make it the industry standard for full-text search. The key takeaways are: Elasticsearch uses inverted indexes for fast text search, with configurable analysis pipelines that control how text is tokenized and normalized.

The Query DSL provides powerful search capabilities including full-text search, filtering, sorting, and highlighting. Aggregations enable real-time analytics and faceted search alongside search results. Start by setting up Elasticsearch with Docker, creating an index with explicit mappings, and building a simple search API. The Elastic documentation at elastic.co/guide is comprehensive and includes practical examples for every feature.

Minh Vo

Slaying code & making it lit fr fr 🔥 tagline