Metabase tips and tricks

2024-10-05T00:00:00+00:00

flexible date bin'ed dashboards</h2>
Your consumers want to view the same data over different timeframes You want to build a dashboard once to prevent definition drifts.
E.g Total Payments by Day, Total Payments by Month, etc
We handle this with a few tricks that the data consumers understand without issue.

Helper Model</h3>

We defined a Helper Model</a> with the time periods we offer for analytics. We're lucky that we don't need super granular stats and can just offer down to hour</code>. You could have custom periods but you will need to replace date_trunc</code> with date_bin</code> in the examples below and take care to read caveat 1.

  </span>SELECT
</span>    time_period
</span>  </span>FROM</span> (
</span>  </span>VALUES
</span>    ('</span>hour</span>'),
</span>    ('</span>day</span>'),
</span>    ('</span>week</span>'),
</span>    ('</span>month</span>'),
</span>    ('</span>quarter</span>'),
</span>    ('</span>year</span>'),
</span>  ) t(time_period)
</span></code></pre>
Filtering the result set</h3>
In any query that requires the ability to date bin we use one of the following as part of the query definition</p>
Use alongside date field filter</h4>
This works great for simple queries where you are filtering on a single date field as the date field filter has a bunch of
useful settings like inclusive/exclusive end, ability to exclude days of the week etc that are often requested by our consumers</p>
SELECT
</span>  date_trunc({{time_period}}, created_at)::</span>date </span>AS time_period,
</span>  </span>SUM</span>(amount)
</span>FROM</span> payments
</span>WHERE
</span>  </span>1</span>=</span>1
</span>  [[AND {{payment_created_at_field_filter}}]]
</span>GROUP BY</span> time_period
</span></code></pre>
When you can't use the field filter</h4>
Metabase doesnt support date field filters on thee virtual tables created by CTE's, and there are times where you want to filter multiple tables to the same time range.</p>
In those cases we add a number of periods</code> filter that defines the how far to look back over instead</p>
SELECT
</span>  date_trunc({{time_period}}, created_at)::</span>date </span>as time_period,
</span>  </span>sum</span>(amount)
</span>FROM</span> payments
</span>JOIN</span> users on </span>payments</span>.</span>user_id </span>= </span>users</span>.</span>id
</span>WHERE
</span>  </span>payments</span>.</span>created_at </span>>= date_trunc({{time_period}}, NOW() - ({{num_periods}} || ' ' || {{time_period}})::interval)
</span>  AND
</span>  </span>users</span>.</span>last_sign_in_at </span>>= date_trunc({{time_period}}, NOW() - ({{num_periods}} || ' ' || {{time_period}})::interval)
</span>GROUP BY</span> time_period
</span></code></pre>
Caveat 1: Quarter</h5>
date_trunc</code> works with quarter</code> but interval</code> doesnt.
It would be a lovely addition1</a></sup> but for now we add a case statement that turns 1 quarter into 3 months. We're Ok with a slight inaccuracy here.
Use a similar technique in you have custom time_periods.
You need to do some math to make the multiplication understandable. e.g num_periods = 4 </code> & time_period = '15 minutes'</code> you want to multiple by 15 before converting to an interval.</p>
payments</span>.</span>created_at </span>>= date_trunc(
</span>  {{time_period}},
</span>  </span>CASE</span> {{time_period}}
</span>    </span>WHEN </span>'</span>quarter</span>' </span>THEN
</span>      date_trunc('</span>quarter</span>', NOW()) - ((({{num_periods}})::</span>integer </span>* </span>3</span>) || '</span> month</span>')::interval
</span>    </span>ELSE
</span>      NOW() - ({{num_periods}} || ' ' || {{time_period}})::interval
</span>  </span>END
</span>)
</span></code></pre>
</p>
Caveat 2: Incomplete periods</h5>
We include data from the incomplete current period but we can change this by adding a less than version to complement each
filter</p>
SELECT
</span>  date_trunc({{time_period}}, created_at)::</span>date </span>AS time_period,
</span>  </span>SUM</span>(amount)
</span>FROM</span> payments
</span>  </span>payments</span>.</span>created_at </span>>= date_trunc({{time_period}}, NOW() - ({{num_periods}} || ' ' || {{time_period}})::interval)
</span>  AND </span>payments</span>.</span>created_at </span>< date_trunc({{time_period}}, NOW())
</span></code></pre>
The majority of our data analysis is done on historic data, but of course you can:</p>

Flip the signs to only look forward</li>
Use it with between and NOW() + num_periods</code> & NOW() - num_periods</code> to look equally in both directions</li>
Define both backward_periods</code> & forward_periods</code> for use in the above if they must be unique</li>
</ul>
Optional targets</h3>
You'll probably have a tracking board, daily or monthly that the leadership team love to watch.
They will also want to compare past periods to the current one without going to a new dashboard. Eg (looking at today vs the big 1 day sale last week)
Combining COALESCE</code> with optional filters and you can inject SQL defaults.</p>
This gets around the fact that raw date filters can't be set to today</code>.</p>
Something like this works</p>
with</span> filtered_payments as (</span>/* some query */</span>)
</span>
</span>SELECT
</span>  </span>COUNT</span>(</span>*</span>)
</span>FROM</span> filtered_payments
</span>WHERE
</span>  date_trunc('</span>day</span>', </span>filtered_payments</span>.</span>created_at</span>) = date_trunc('</span>day</span>', COALESCE( [[{{date_to_look_at}} ,]] NOW() ))
</span></code></pre>
Percent function</h3>
To remove some error prone casting and coalesce'ing related to calculating
percentages, especially over counts, we added a function to make it more
readable</p>
CREATE OR REPLACE FUNCTION </span>public.</span>percent</span>(portion </span>numeric</span>, total </span>numeric</span>, round_level </span>integer DEFAULT </span>1</span>)
</span>  RETURNS </span>numeric
</span>  LANGUAGE sql
</span>  IMMUTABLE PARALLEL SAFE STRICT
</span>AS $function$
</span>    </span>select
</span>        </span>CASE
</span>        </span>WHEN</span> total = </span>0 </span>THEN </span>0
</span>        </span>ELSE</span> ROUND(</span>100 </span>*</span> (portion / total), round_level)
</span>        </span>END
</span>$function$
</span></code></pre>
It replaces the following, while also handling counts that return 0</code></p>
SELECT
</span>  ROUND(100 * cast(count(*) FILTER (where amount > 100) as float)/cast(count(*) as float)) AS old_big_payment_rate,
</span>  percent(
</span>    COUNT(*) FILTER (where amount > 100),
</span>    COUNT(*)
</span>  ) AS new_big_payment_rate
</span>FROM payments
</span></code></pre>
Commonly accessed questions</h3>
We do periodic cleaning of the available dashboards; we'll archive things that havent been accessed in the last 6-12 months to keep the number of questions and dashboards down.
The below will give you the commonly accessed queries, flipping or changing the order to last_execution will show you the ones that aren't often accessed.
Use an Anti-Join to find the reports that haven't been run at all in the period you're checking.</p>
with card_data as (
</span>  select
</span>    card_id,
</span>    COUNT(*) as execution_count,
</span>    MAX(started_at) as last_execution
</span>  from query_execution
</span>  where
</span>    started_at > NOW() - '12 months'::interval
</span>  GROUP BY 1
</span>)
</span>
</span>select
</span>  report_card.id,
</span>  report_card.name,
</span>  CONCAT('https://data.example.com/questions/', report_card.id),
</span>  card_data.execution_count,
</span>  card_data.last_execution
</span>from report_card
</span>join card_data on card_data.card_id = report_card.id
</span>where
</span>  report_card.archived = false
</span>ORDER BY card_data.execution_count DESC
</span></code></pre>
^{1</sup>
Wish: support "quarter" in Interval</a></p>
</div>}

Using certificate based ssh authentication

2014-09-20T00:00:00+00:00

What is a Certificate?</h2>
A certificate is a form of Public Key Cryptography</a> that allows you to trust someone.
The certificate authority's (CA) private key is used to sign the public key of anyone or anything that is trusted by the CA.
Anybody can then decide to trust the same people as the CA by adding the CA's public key to their keyring.

Why should I use one?</h2>
One of the main reasons is that CA signing works in both directions.
A user can check that a machine is trusted instead of relying on adding individual hosts to `~/.ssh/known_hosts</code>.`
`This is useful if you are using the same IP address but you are recreating a machine for testing purposes and you don't want to be bugged by SSH key checking messages.`
`A machine can check that a user is trusted instead of relying on adding user's public key's to ~/.ssh/authorized_keys</code> for each user that should be allowed to login.`
`Signing keys also allows very fine grained control over many aspects that key based authentication doesn't.`
`The reason I have switched from key based to cert based authentication is that it is much easier to handle access controls access your personal machines when you are regularly creating new VM's and VPS's.`
`You only need to add the CA's public key to a server rather than all of your user's public keys to the correct ~/.ssh/authorized_key</code>s file.`
`Setting it up.</h2> For this you will need:`2+ machines running your preferred form of Linux (I used Debian 7)</li> Openssh 5.4+</li> Basic knowledge of the command line</li> </ul> In this example I will be using the machine I am writing this on as both the CA and the user I want to trust. This isn't totally secure but will work perfectly fine with a small number of users. First we will create a user CA which will allow the users to sign into servers without a prompt. The first step is to create your CA's keypair. $ mkdir -p ~/.ssh/ca $ cd ~/.ssh/ca $ ssh-keygen -f ca -C "Comment on the keys" Generating public/private rsa key pair. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in ca. Your public key has been saved in ca.pub. $ ls ca ca.pub </code></pre> You should provide a strong passphrase when asked as this will mean that if attacker gains access to your keys your security is not compromised. The next step is to place the public key on the machine(s) that should trust the users. $ scp ca.pub user@example.com:/home/user $ ssh user@example.com "sudo cp ~/ca.pub /etc/ssh" $ ssh user@example.com "sudo sed -i '/TrustedUserCAKeys/d' /etc/ssh/sshd_config" $ ssh user@example.com "echo 'TrustedUserCAKeys /etc/ssh/ca.pub' | sudo tee -a /etc/ssh/sshd_config" $ ssh user@example.com "sudo /etc/init.d/ssh restart" # The above will only work on a machine set up with passwordless sudo # Otherwise ssh to the machine and run the commands manually </code></pre> This sets up the server to trust signed keys from the CA, so the next step is to sign our users public key. If you already have a set of user keys you should skip the first step. $ ssh-keygen Generating public/private rsa key pair. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in ~/.ssh/id_rsa. Your public key has been saved in ~/.ssh/id_rsa.pub # This generates a keypair with default settings # Like with the CA keypair you should use a strong password for # added security. $ mkdir signed_users $ cd signed_users $ cp ~/.ssh/id_rsa.pub user.pub $ ssh-keygen -s ~/.ssh/ca/ca -I "Comment" -n "user" user.pub Enter passphrase: # Enter the passphrase you used to create the ca keypair # You can add more than one user by comma separating them # You will only be allowed to login as the users specified # in this list and it is required $ chmod 600 user-cert.pub # Set the correct permissions on your signed key $ ls user.pub user-cert.pub $ cp user-cert.pub ~/.ssh/id_rsa-cert.pub </code></pre> You should keep a copy of user.pub safe because if you ever want to revoke a users trusted status you need to know their public key. You can easily repeat this for each machine you use regularly. If you use an automation tool</a> to set up new machines it is simple to set up a task that will create and sign a keypair allowing you to log in to any of your machines from it straight away. There are many other options</a> you can add to the signed key to restrict and secure the user further. Some examples are: Setting an expiration on the signed key</li> Only allowing the key to be used from certain address's</li> Forcing a specific command to be run instead of a shell when using the certificate</li> </ul> Revoking users</h2> If you lose access to a set of user keys it is fairly easy to revoke their access to your machines. $ ssh user@example.com "[ -f /etc/ssh/ssh_revoked_keys ] || sudo install -m 644 -o root -g root /etc/ssh/ssh_revoked_keys" $ cat user_to_revoke.pub | ssh user@example.com "sudo tee -a /etc/ssh/ssh_revoked_keys" $ ssh user@example.com "sudo sed -i '/RevokedKeys/d' /etc/ssh/sshd_config" $ ssh user@example.com "echo 'RevokedKeys /etc/ssh/ssh_revoked_keys' | sudo tee -a /etc/ssh/sshd_config" $ ssh user@example.com "sudo /etc/init.d/ssh restart" # Again this requires passwordless sudo </code></pre> This does have to be run on each of your machines but it is still less work than combing through authorized_keys files for each user account on every machine you have. Final notes.</h2> Keep you CA private key safe. This can be used to sign new keys that can then access your machines.</li> Use passphrase's and a SSH agent. These improve security so that an attacker must have access to a key and know the passphrase before using it. Using an SSH agent means you only have to input your passphrase once across machine reboots.</li></li> You should try to follow the principle of least privilege</a> for user accounts and give each one their own signed key.</li> </ul>

Daniel Wakefield

Metabase tips and tricks

Filtering the result set</h3> In any query that requires the ability to date bin we use one of the following as part of the query definition</p>

Use alongside date field filter</h4> This works great for simple queries where you are filtering on a single date field as the date field filter has a bunch of useful settings like inclusive/exclusive end, ability to exclude days of the week etc that are often requested by our consumers</p>

Caveat 2: Incomplete periods</h5> We include data from the incomplete current period but we can change this by adding a less than version to complement each filter</p>

Percent function</h3> To remove some error prone casting and coalesce'ing related to calculating percentages, especially over counts, we added a function to make it more readable</p>

Using certificate based ssh authentication

Setting it up.</h2> For this you will need:</p> 2+ machines running your preferred form of Linux (I used Debian 7)</li> Openssh 5.4+</li>

Revoking users</h2> If you lose access to a set of user keys it is fairly easy to revoke their access to your machines.</p>

Filtering the result set</h3>
In any query that requires the ability to date bin we use one of the following as part of the query definition</p>