PostHog Handbook Library / Engineering

1,129 words. Estimated reading time: 6 min.

# Query performance

This document goes over:

- What tools are available to understand and measure query performance
- Importance of page cache
- General tips and tricks for performant queries

### Tooling

#### clickhouse-client

`clickhouse-client` is a command-line application for running queries against ClickHouse.

When executing queries, it details progress, execution time, how many rows and gigabytes of data were processed, and how much CPU was used.

Image: clickhouse-client progress reporting

You can get additional logging from ClickHouse by setting `SET send_logs_level = 'trace'` before running a query.

#### `system.query_log`

ClickHouse saves all queries it runs into `system.query_log` table.

It includes information on:

- What query was run and when
- How long did it take to execute
- How many resources did it take up: memory, rows/bytes read
- In case of errors, exception information

Some tips for querying the `query_log`:

- For distributed queries, filter by `is_initial_query` to disambiguate distributed queries i.e. `is_initial_query = 1` denotes the query sent to the coordinator node, whereas `is_initial_query = 0` denotes internally-generated queries sent to gather data from other shards
- Use `clusterAllReplicas('<cluster_name>', system.query_log)` to query all nodes
- Filter on `type = 'QueryFinish'` if you only want data on queries that completed successfully. Forgetting to do so can skew averages since the results will include `QueryStart` events, which have columns such as `query_duration_ms` set to 0.
- `ProfileEvents` column contains a lot of useful performance data on each query, not all of which is documented. Check the source for a full list all measurements.

At PostHog, we also add metadata to each query via the `log_comment` setting to make results easier to analyze. This includes information on the source of the query and how it was constructed. See this runbook for more details.

An example query to get recent slow queries:

    SELECT
        query_duration_ms,
        query,
        event_time,
        read_rows,
        formatReadableSize(read_bytes) as read_size,
        result_rows,
        formatReadableSize(result_bytes) as result_size,
        formatReadableSize(memory_usage) as memory,
        columns,
        query_id
    FROM system.query_log
    WHERE
        query NOT LIKE '%query_log%'
        AND type = 'QueryFinish'
        AND event_time > now() - toIntervalDay(3)
        AND query_duration_ms > 30000
        /* If using log_comment, consider including something like:
        AND JSONExtract(log_comment, 'kind') = 'request'
        */
    ORDER BY query_duration_ms desc
    LIMIT 10

Note that this table is not distributed - on a cluster setting you might need to run query against each node separately or do ad-hoc distributed queries.

#### `EXPLAIN`

Previous pages in this manual showed various examples of using the ClickHouse `EXPLAIN` statement to your advantage.

Various forms of explain can detail:

- If and how much data ClickHouse was able to avoid processing thanks to schema setup
- If and how ClickHouse "optimizes" the query by moving columns to `PREWHERE`
- Detailing how the query is planned to be executed

Read more about `EXPLAIN` in ClickHouse's EXPLAIN Statement docs.

#### Flame graphs

For CPU-bound calculations, flamegraphs can help visualize what ClickHouse worked on during query execution.

<div class="relative mt-2 mb-4"

<object data={'/images/flamegraph.svg'} type="image/svg+xml" />

We've built flamegraph support into PostHog. You can find tools to generate flamegraphs for queries under PostHog instance settings.

### Importance of the page cache

When running queries, you might encounter an odd artifact: the first time you run a query, it's really slow but it speeds up significantly when run again.

This behavior is due to the Linux page cache. In broad terms, the operating system caches recently read files into memory, speeding up subsequent reads of the same data.

As most queries in ClickHouse are dependent on fast I/O to execute fast, this can have a significant effect on query performance. It is a reason why at PostHog our ClickHouse nodes have a lot of memory available.

#### Effect on benchmarking

This behavior can be a problem for profiling: users constructing new queries might not hit the page cache and receive a worse experience than benchmarking may show.

This means it's often important to wipe page cache on ClickHouse when doing queries. This can be achieved with the following command on a ClickHouse node:

    sudo sh -c "/usr/bin/echo 3 > /proc/sys/vm/drop_caches"

> Note that the above will only drop the cache on the given node, but distributed queries might still be affected from the page cache on nodes in the other shards.

For completely clean benchmarking, you might also want to drop ClickHouse's internal mark cache.

You can also use the `min_bytes_to_use_direct_io` setting to bypass the page cache at the query level.

When set to a value greater than 0, ClickHouse will use `O_DIRECT` for disk reads whenever the total data to be read exceeds the threshold (in bytes).

    SETTINGS min_bytes_to_use_direct_io = 1;

### Join algorithms

JOINs are expensive in ClickHouse, so any opportunities to speed them up are welcome.

One of the quickest possible wins on that front is by benchmarking different join algorithms.

Newer ClickHouse versions have added more algorithms, and it's worth keeping an eye on the ones that come out and check if they help improve query performance.

In PostHog's case, we have moved away from the `default` algorithm (alias for `direct,hash`) in favor of `direct,parallel_hash`. `parallel_hash` is effectively the same as `hash`, but it does the computation in multiple buckets. It aims to be faster by consuming a bit more resources.

In our extensive benchmarking (including in our production environment), we've found that across the board using `parallel_hash` over `hash` provided us with the following speed improvements:

- Average: parallel_hash was 1.23x faster
- p95: parallel_hash was 1.49x faster
- p99: parallel_hash was 1.37x faster

This came at a cost of up to 1.5x more memory usage, as well as a bit more CPU usage, which were acceptable tradeoffs in our case.

### Tips for achieving well-performing queries

Previous pages in the ClickHouse manual have highlighted the importance of setting up the correct schema and detailing how queries work in a distributed setting.

This section highlights some general rules of thumb that can help speed up queries:

- Make sure query is supported by ORDER BY. This is frequently the single biggest factor in query performance.
- Avoid JOINs and sub-queries whenever possible. Denormalizing data can help here.
- Partition data so old partitions can be skipped.
- Push down any and all WHERE conditions to in queries as deep as possible, as ClickHouse won't do it for you.
- Avoid reading columns that are not needed for the query.
- Test if moving small column filters to `PREWHERE` helps performance.
- Avoid reading large columns in `PREWHERE`.
- Compress columns with appropriate codecs.
- Make use of the extensive catalogue aggregation functions available in ClickHouse.
- Looking up single rows is frequently slow.
- ORDER BY and LIMIT/OFFSET can be slow in sharded settings.
- Cheat: preaggregate data or leverage materialized columns to pull out data from large columns.
- Leverage query settings in special cases.
- Benchmark different join algorithms on queries that use JOINs. From our testing, `parallel_hash` can help speed up most types of queries with JOINs at the expense of a bit more memory and CPU usage. `grace_hash` is also a promising new algorithm introduced in ClickHouse 22.12.

Also always do your benchmarking in a realistic setting: on large datasets on powerful machines.

Next in the ClickHouse manual: Operations

Canonical URL: https://posthog.com/handbook/engineering/clickhouse/performance

GitHub source: contents/handbook/engineering/clickhouse/performance.mdx

Content hash: `1a82927a7d087715`