Category Archives: Uncategorized

Shopify app with Rust: Part 6 – request shop data

The previous post (get OAuth access token) covered retrieving the OAuth access token that can be used for persistent permission to retrieve a shop’s data. To make use of that in the context of an application, we’ll need to attach it to a request. There are a lot of small supporting pieces here, so you’ll need a bit of patience.

First things first, if you’re actually coding along then when you open the application within Shopify’s dashboard, you’re likely seeing something disappointingly like:

Firefox can’t open this page

This is a security feature that is enabled by Rocket by default that prevents our application from being embedded within another webpage (e.g. the Shopify partner dashboard). Specifically, this is the Rocket shield. There are still some pieces of Rocket that are unimplemented, e.g. support for Content-Security-Policy, so let’s disable some protection entirely to make it embeddable!

// main.rs

use rocket::{
    ...
    shield::{Frame, Shield},
    ...
};
...

fn rocket() -> rocket::Rocket<rocket::Build> {
    rocket::build()
        .mount("/", routes![install, callback, index])
        .attach(Shield::default().disable::<Frame>())
}
...

With the X-Frame-Options header removed (via Frame), the page should now be embeddable within the Shopify portal (not that we have much to show…).

Making a really good and not at all terrible UI

Aligned with the theme of this series, lets just barrel forwards naively and fix stuff when it doesn’t work. Any self respecting application needs some form of UI, so far we’re lacking that regard. Rather than add that logic in our /install or /callback request handlers, lets add a new API. As these blog posts are not about UI technologies, we’re going to do about the laziest possible thing and return static HTML directly with no supporting tooling. We’ll spice it up just a tiny bit by making it parameterizable with the shop’s name:

// main.rs

fn build_index_html(shop: &str) -> String {
	format!(
		r#"<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>R'al cool Shopify app</title>
  <script>
    function doFancyFrontEndThings() {{
      window.alert("Ya dun clicked the button");
    }}
  </script>
</head>
<body>
  <h1>Welcome to this Shopify Application, {}</h1>
  <button onclick="doFancyFrontEndThings()">Click me!</button>
</body>
</html>"#,
		shop,
	)
}

Add a new route that returns this page so that we can see our really rockin’ UI:

// main.rs
use rocket::{
    response::{content, Redirect},
    ...

#[rocket::get("/")]
pub async fn index() -> Result<content::Html<String>, &'static str> {
    Ok(content::Html(build_index_html("user"))
}
...

fn rocket() -> rocket::Rocket<rocket::Build> {
    rocket::build()
        .mount("/", routes![install, callback, index])
        ...

We also need a mechanism to see this page, so how about we retool our /callback to redirect here:

// main.rs
use rocket::{
    ...
    uri, State,
};
...

#[rocket::get("/callback?<code>&<shop>")]
pub async fn callback(
    code: &str,
    shop: &str,
    installations: &State<Installations>,
) -> Result<Redirect, &'static str> {
    ...
    Ok(Redirect::to(uri!(index())))
}

Now if we visit the app again we should actually be able to see HTML, which is dramatically cooler than the terribleness we had previously:

Glorious rendered HTML!

Feels like we’re getting close… an installable application with a UI?! The big piece that’s missing is getting Shopify data.

Authorized request to Shopify

Currently we have the access token for our test shop being printed out into the application server logs (bad practice!), so lets use it. Run the application and save the access token somewhere, we’ll hardcode it shortly. Shopify has a number of different APIs and a mix off REST based and GraphQL based interfaces. For some extra complexity, there’s not feature parity between the two. In general, it seems the GraphQL version is intended to be the “future”, so we’ll go down that route. Most APIs relevant for simple apps will come from the Admin API section.

If you’re trying to craft your own query, the Shopify GraphiQL explorer is a great tool to use. If you were doing this for real, you’d likely use a GraphQL client in Rust (e.g. graphql_client), but to keep our dependencies as minimal as possible we’re going to cut some corners. We will add the serde_json crate though, as a halfway between unstructured text and full deserialized structs.

// Cargo.toml
...
[dependencies]
rocket = "0.5.0-rc.1"
reqwest = { version = "0.11", features = ["json"] }
serde = "1"
serde_json = "1"

As we’re not trying to tackle “how to work with Shopify’s API in general”, lets use retrieving the shop’s name as placeholder logic. To do so, we’ll need to form a GraqhQL request and submit it to the right endpoint. Well need 4 pieces to build this HTTP POST:

  1. the endpoint: https://{shop}.myshopify.com/admin/api/2021-07/graphql.json
  2. the Content-Type header: Content-Type: application/graphql
  3. the shop’s access token as a header: X-Shopify-Access-Token: {token}
  4. The actual GraphQL query

As with the HTML, make a constant for the very simple GraphQL query. In a real app instead of getting the shop’s name you might get recent orders or customers:

// main.rs

const QUERY: &str = r#"
{
  shop {
    name
  }
}"#;

Then build and submit the request. Note that the access token is a password and as such should not be shared. This has red flags all over it.

//main.rs
use reqwest::header::CONTENT_TYPE;
...
const APPLICATION_GRAPHQL: &str = "application/graphql";
const SHOPIFY_ACCESS_TOKEN: &str = "X-Shopify-Access-Token";

#[rocket::get("/")]
pub async fn index() -> Result<content::Html<&'static str>, &'static str> {
	let admin_api_uri = format!(
		"https://{}.myshopify.com/admin/api/2021-07/graphql.json",
		"toby-test-store2"
	);

	let client = reqwest::Client::new();
	let response = client
		.post(admin_api_uri)
		.header(CONTENT_TYPE, APPLICATION_GRAPHQL)
		.header(
			SHOPIFY_ACCESS_TOKEN,
			"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",
		)
		.body(QUERY)
		.send()
		.await
		.map_err(|_| "Error sending request")?;

	let response_body: serde_json::Value = response
		.json()
		.await
		.map_err(|_| "Error getting response body as json")?;

	let pretty_json = serde_json::to_string_pretty(&response_body).map_err(|_| "Error prettifying JSON")?;

	println!("Response is: {:#?}", pretty_json);
	Ok(content::Html(HTML))
}
...

When I run this, I get a response that looks something like:

{
  "data": {
    "shop": {
      "name": "Toby Test Store2"
    }
  },
  "extensions": {
    "cost": {
      "actualQueryCost": 1,
      "requestedQueryCost": 1,
      "throttleStatus": {
        "currentlyAvailable": 999,
        "maximumAvailable": 1000.0,
        "restoreRate": 50.0
      }
    }
  }
}

If we traverse this JSON object with wild abandon, we can update our UI so it’s outrageously personalized:

// main.rs
...
	let response_body: serde_json::Value = response
		.json()
		.await
		.map_err(|_| "Error getting response body as json")?;

	let shop_name = &response_body["data"]["shop"]["name"].as_str().unwrap();

	Ok(content::Html(build_index_html(shop_name)))
}
...
Whoa – data from the API in our UI!

Status

Where we started in this post: We had an installable application that would solicit an OAuth access token, but not do anything with it

Where we ended with this post: We tied up a couple loose ends, built a dynamite UI, made a request with the OAuth access token, and served up a UI with data retrieved from Shopify. Things are looking particularly legit around these parts.

Next post: We’ll avoid hard coding the very secret OAuth access token in the browser and store it in memory in Rocket instead

Shopify app with Rust: Part 5 – get OAuth access token

The last post (starting OAuth) tackled the first pass at the app installation API that serves Shopify’s OAuth flow. It also covered declaring the redirect URI, so Shopify sends the application server a web request, the server responds with a redirect to Shopify to authorize the installation, and then Shopify sends the user back to the application server.

A quick refresher on Shopify’s described OAuth flow:

1. The merchant makes a request to install the app.
2. The app redirects to Shopify to load the OAuth grant screen and requests the required scopes.
3. Shopify displays a prompt to the merchant to give authorization to the app, and prompts the merchant to log in if required.
4. The merchant consents to the scopes and is redirected to the redirect_uri.
5. The app makes an access token request to Shopify including the client_id, client_secret, and code.
6. Shopify returns the access token and requested scopes.
7. The app uses the access token to make requests to the Shopify API.
8. Shopify returns the requested data.

https://shopify.dev/apps/auth/oauth#the-oauth-flow

We have an authorized request knocking at the door of the application server (/callback API) but we’re not doing anything with it. In this post we’re going to tackle the body of:

// main.rs

#[rocket::get("/callback")]
pub fn callback() -> &'static str {
	"Now what..."
}

The goal here is to address: “The app makes an access token request to Shopify including the client_id, client_secret, and code.” If you take a close look at the output of the previous attempt to install an application, it seems we’ve missed quite a few parameters on the callback API:

Callback API invocation

Specifically on:

GET /callback?code=8a99f98c7deb52660f76219dee5ada50&hmac=6cb4c00bf66110d41cb4d7e26c59ade80d3b322de7ca0f28e54ac9b8dace4f4b&host=dG9ieS10ZXN0LXN0b3JlMi5teXNob3BpZnkuY29tL2FkbWlu&shop=toby-test-store2.myshopify.com&state=random-value&timestamp=1639244833 text/html

There are 6 query parameters!

  • code: The authorization code provided in the redirect, this is what we were after in the previous step and will use in the next step to get a token!
    • looks like: 8a99f98c7deb52660f76219dee5ada50
  • hmac: This is a signature we’ll use later to verify the request is unaltered from Shopify, it’s a bit ol’ TO-DO right now though
    • looks like: 6cb4c00bf66110d41cb4d7e26c59ade80d3b322de7ca0f28e54ac9b8dace4f4b
  • host: base64 encoded host name appended with the /admin URL segment
    • looks like dG9ieS10ZXN0LXN0b3JlMi5teXNob3BpZnkuY29tL2FkbWlu
    • decoded, looks like: toby-test-store2.myshopify.com/admin
  • shop: The plaintext name of the merchant’s shop (without trailing /admin)
    • looks like: toby-test-store2.myshopify.com
  • state: The nonce we provided in the installation redirect URL
    • looks like: random-value right now, but we’ll fix that eventually…
  • timestamp: seconds since epoch of the request, I think?
    • looks like: 1639244833

The timestamp, state, and hmac we can ignore entirely for the time being (this is a development app, we can’t ignore these in production). The host is useful, but not immediately, so let’s ignore it for the time being as well. The code and shop are exactly what we need for the next step. First, let’s make them accessible in our request handler:

// main.rs
#[rocket::get("/callback?<code>&<shop>")]
pub fn callback(code: &str, shop: &str) -> &'static str {
    println!("The code is: {}", code);
    println!("The shop is: {}", shop);
    "Now what..."
}

Now, what are we ultimately doing with these values? We have a code that represents the Shopify store’s agreement to install the application with the requested permissions. Now that we have that code, we’re going back to Shopify to say “Look, the shop agreed – can we access their data?”. This happens by exchanging the authorization code for an access token. The access token will let us make requests to Shopify that are authorized to access the granting store’s data. Before we dive into that too much, lets actually get an access token.

We need the application server to make a request. To do so, we’ll need an HTTP client. Looking at lib.rs, reqsounds godwest seems like it may be overkill but will likely do the job. We’re using virtually no features of the HTTP client library, and in this case it has to be synchronous regardless of whether the underlying code is sync or async.

Add reqwest to your Cargo.toml:

[package]
name = "template"
version = "0.1.0"
edition = "2021"

[dependencies]
rocket = "0.5.0-rc.1"
reqwest = { version = "0.11" }

Now let’s send an HTTP POST to https://{shop}/admin/oauth/access_token. In the spirit of just doing stuff, let’s give it a shot and see what happens. We’ll ignore Reqwest’s advice to share a client for the time being (another TO-DO?) and go for it. Lets also take the opportunity to change the return type to a Result just because it makes the code a bit easier to work with.

// main.rs
#[rocket::get("/callback?<code>&<shop>")]
pub async fn callback(code: &str, shop: &str) -> Result<&'static str, &'static str> {
    println!("The code is: {}", code);
    println!("The shop is: {}", shop);

    let access_token_uri = format!("https://{}/admin/oauth/access_token", shop);

    let client = reqwest::Client::new();
    let response = client
        .post(access_token_uri)
        .send()
        .await
        .map_err(|_| "Error sending request")?;

    println!("Response status: {}", response.status());

    let response_body = response
        .text()
        .await
        .map_err(|_| "Error getting response body as text")?;

    println!("{}", response_body);

    Ok("Now what...")
}

Some good things: the request itself succeeds in sending and a response comes back. Areas for improvement, the request yields an HTTP 400 Bad Request and the body is HTML. If you render the HTML, it looks something like this:

HTTP 400 Bad Request

As it says on Shopify, this is a JSON body with 3 properties:

  • client_id: this is our application’s API key, coming from the Shopify partner dashboard for the app
  • client_secret: this is our applications API secret key, also coming from the Shopify partner dashboard for the app
  • code: this is the code query parameter from the callback URL we have is populated

So, lets grab the values and try again! client_id and client_secret come from the Shopify partner dashboard:

This is where API key and API Secret key are coming from

The code is a query parameter in the original request. We’re crafting a JSON request, and basically as soon as anyone mentions JSON it’s time to add the Serde crate. While we’re there, we’ll enable the json feature of Reqwest so we can use Serde:

[package]
name = "template"
version = "0.1.0"
edition = "2021"

[dependencies]
rocket = "0.5.0-rc.1"
reqwest = { version = "0.11", features = ["json"] }
serde = "1"

One of the easiest ways I’ve found to craft a JSON request is to define a type for the request body and then let Serde serialize it to JSON. Adding the appropriate import and a stuct definition:

// main.rs
use rocket::{response::Redirect, routes, serde::Serialize};

#[derive(Serialize)]
struct AccessTokenRequest<'a> {
    client_id: &'a str,
    client_secret: &'a str,
    code: &'a str,
}

Now we can use that and amend our request so it contains the correct body:

#[rocket::get("/callback?<code>&<shop>")]
pub async fn callback(code: &str, shop: &str) -> Result<&'static str, &'static str> {
    let access_token_uri = format!("https://{}/admin/oauth/access_token", shop);

    let body = AccessTokenRequest {
        client_id: "801f137828580f71e33ad6b14f75e533",
        client_secret: "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",
        code,
    };

    let client = reqwest::Client::new();
    let response = client
        .post(access_token_uri)
        .json(&body)
        .send()
        .await
        .map_err(|_| "Error sending request")?;

    println!("Response status: {}", response.status());

    let response_body = response
        .text()
        .await
        .map_err(|_| "Error getting response body as text")?;

    println!("{}", response_body);

    Ok("Now what...")
}

Start up the application server, try installing or opening the app and:

Voilà! An access token!

We have an access token for the shop! As a side note, this is effectively a password that can be used to access a store’s data. Never reveal the value, and be diligent to not let it show up in screenshots in a blog post or similar.

As a quality of life improvement, let’s also deserialize the response body so it’s easier to use. As a quirk of the deserialization, we can only use owned types, so we need String instead of slices:

// main.rs
use rocket::{
    response::Redirect,
    routes,
    serde::{Deserialize, Serialize},
};

#[derive(Deserialize, Debug)]
struct AccessTokenResponse {
    access_token: String,
    scope: String,
}

#[rocket::get("/callback?<code>&<shop>")]
pub async fn callback(code: &str, shop: &str) -> Result<&'static str, &'static str> {
    ...
    let response_body: AccessTokenResponse = response
        .json()
        .await
        .map_err(|_| "Error getting response body as JSON")?;

    println!("{:?}", response_body);
    ...

Once again, referring back to the Shopify OAuth flow we’ve now tackled two more steps:

1. The merchant makes a request to install the app.
2. The app redirects to Shopify to load the OAuth grant screen and requests the required scopes.
3. Shopify displays a prompt to the merchant to give authorization to the app, and prompts the merchant to log in if required.
4. The merchant consents to the scopes and is redirected to the redirect_uri.
5. The app makes an access token request to Shopify including the client_id, client_secret, and code.
6. Shopify returns the access token and requested scopes.
7. The app uses the access token to make requests to the Shopify API.
8. Shopify returns the requested data.

https://shopify.dev/apps/auth/oauth#the-oauth-flow

Status

Where we started in this post: We had an app that “installable” and we could both request and be granted permissions for a store installing the app. Unfortunately we could not leverage those permissions to do anything

Where we ended with this post: We’ve exchanged the ephemeral authorization a store grants us for a permanent authorization. In practice this looks like using the provided code value for an OAuth access token. With this token, we can (theoretically at this point) make requests to Shopify for data that belongs to the associated store

Next post: Make an authorized request and obtain a shop’s data!

Shopify app with Rust: Part 4 – Get OAuth Authorization Code

Previously (the app server) we got to the point where a Shopify store owner could try to install an application, they would make it to our server, our server would do some work, and then the user would get sent to oblivion. Now we’re going to try and correct that, send the user somewhere slightly more useful.

For the time being, we’re going to completely ignore both the HMAC validation and the nonce. They’re important, but for the sake of satisfaction checking things off they can be deferred (can’t go to production without it though). The next step is to show the application permissions prompt. This is pretty straightforward, we just need to collect a handful of values and then redirect the Shopify application:

  • shop: the name of the shop installing the app, looks something like shop-name.myshopify.com
  • API key: the API key of the application (not the API secret key), available in the partner portal – looks something like c2ecfe6c6b3745cfaaa9a539b1dfe8a7
  • scopes: a comma separated list of all of the permissions the app will have. A possible value might be something like: read_products,read_product_listings,read_orders,read_customers
  • redirect URI: where you want the user to go after everything is said and done, e.g. the landing page for your application
  • nonce: a random value that can be checked at the next step to ensure the integrity of the process – can be anything, e.g. a UUID works great
  • access mode: the lifetime of the the token you’ll ultimately get that will allow you to do the things specified in scopes on the user’s behalf. This has to be either per-user or value

Collecting all the values takes a bit, but once you’ve done so the next step is very straightforward. You jam them into a a new URL and this is ultimately what we redirect to:

https://{shop}/admin/oauth/authorize?client_id={client_id}&scope={scopes}&redirect_uri={redirect_uri}&state={nonce}&grant_options[]={access_mode}

Implementing this is as straightforward as it seems – amend the previously defined route to redirect somewhere useful:

#[allow(unused_variables)]
#[rocket::get("/install?<hmac>&<shop>&<timestamp>")]
pub fn install(hmac: &str, shop: &str, timestamp: &str) -> Redirect {
    let api_key = "801f137828580f71e33ad6b14f75e533"; // Obviously don't use _MY_ key, swap this out for your own
    let scopes = "read_products,read_product_listings,read_orders,read_customers"; // These will be custom for you depending on your goals
    let nonce = "random-value"; // TODO: This should be unique to each OAuth flow, so hold your nose for a bit...

    let redirect = format!("https://{shop}/admin/oauth/authorize?client_id={client_id}&scope={scopes}&redirect_uri={redirect_uri}&state={nonce}&grant_options[]={access_mode}",
		shop = shop,
		client_id = api_key,
		scopes = scopes,
		redirect_uri = "this-does-not-exist-yet", // Redirecting to oblivion
		nonce = nonce,
		access_mode = "value");

    Redirect::to(redirect)
}

Now, try the install dance again and see what we get!

“The redirect_uri is not whitelisted”

Now isn’t that interesting, we redirected somewhere but something else is going on! We’re one step further and have uncovered a brand new way to screw this up. We redirected the user, but in the very beginning we just filled in the field in our app’s configuration with nonsense. Time to go back and fill in a value that lines up with the app. I used “this-does-not-exist-yet” in the code above, so it’s appropriate to be skeptical this will solve all our problems, but lets see.

In the partner portal, click on your app and then “App setup”, “find Allowed redirection URL(s)” and add the value we’re redirecting to (“this-does-not-exist-yet”), then save.

Fails validation

Looks like the value we’ve used is too fake to be useful! Update the redirect_uri in the code to something that looks more like a URL (e.g. https://this-does-not-exist-yet.test) and put the corresponding value in the configured “Allowed redirection URL(s)”.

    let redirect = format!("https://{shop}/admin/oauth/authorize?client_id={client_id}&scope={scopes}&redirect_uri={redirect_uri}&state={nonce}&grant_options[]={access_mode}",
		shop = shop,
		client_id = api_key,
		scopes = scopes,
		redirect_uri = "https://this-does-not-exist-yet.test", // <-- This is what changed here
		nonce = nonce,
		access_mode = "value");

Now restart your server and do the installation dance again:

Made it to the permissions page

We’ve snuck by the validation! We’re so close to having an installable Shopify application, just click that sweet, sweet “Install unlisted app” button and…

We’ve redirected to oblivion again

I have to be honest, I saw this one coming. Obviously the value we used for redirect_uri was a bit less useless. We’ll need our application to have at least one more route, so we can redirect users there at this point. Lets fix this quick:

  1. add a brain-dead API, e.g. I chose (/callback) that will return some static text (just so we have some response)
  2. mount the API
  3. update the redirect_uri in your code so to reflect this new value
  4. re-re-update the “Allowed redirection URL(s)” in your Shopify partner portal’s app configuration to be whatever your real value is for the API we just introduce
    let redirect = format!("https://{shop}/admin/oauth/authorize?client_id={client_id}&scope={scopes}&redirect_uri={redirect_uri}&state={nonce}&grant_options[]={access_mode}",
		shop = shop,
		client_id = api_key,
		scopes = scopes,
		redirect_uri = "https://2f51-204-237-50-186.ngrok.io/callback", // <-- Updated to be real!
		nonce = nonce,
		access_mode = "value");

#[rocket::get("/callback")]
pub fn callback() -> &'static str {
	"Now what..."
}

fn rocket() -> rocket::Rocket<rocket::Build> {
    rocket::build().mount("/", routes![install, callback])
}

For me, the whole URL for this new callback API looks like: https://2f51-204-237-50-186.ngrok.io/callback. Note that because this is using ngrok, every time you re-establish an ngrok connection you’ll have to update this value and keep it in sync between Shopify configuration and your application code, lest you run into the errors we saw above. This means it’s easiest to keep the ngrok application running the whole time you’re developing so you don’t have to go whitelist the address in your “Allowed Redirection URL(s)” field in your Shopify app configuration constantly.

Shockingly enough, at this point when we try again we will technically have installed your application into the store. To test again you can delete it if you’d like, but we’ll end up the same place if you try and reinstall it in the same store, so dealer’s choice.

After making the code and configuration changes, restart the server, reinstall the application, annnnnddd…..

Successful redirection!

Looking at the terminal:

terminal output

We can see a request to:
/callback?code=94872a044567b4711e544497b7ce2afc&hmac=14dd56d06cd2962b13cc833dc2472173eeba0223135cdd0b99f5fc45039755ee&host=dG9ieS10ZXN0LXN0b3JlMi5teXNob3BpZnkuY29tL2FkbWlu&shop=toby-test-store2.myshopify.com&state=random-value&timestamp=1633314364

That’s a lot of stuff that we’re currently not doing anything with. Revisiting Shopify’s described OAuth flow:

1. The merchant makes a request to install the app.
2. The app redirects to Shopify to load the OAuth grant screen and requests the required scopes.
3. Shopify displays a prompt to the merchant to give authorization to the app, and prompts the merchant to log in if required.
4. The merchant consents to the scopes and is redirected to the redirect_uri.
5. The app makes an access token request to Shopify including the client_id, client_secret, and code.
6. Shopify returns the access token and requested scopes.
7. The app uses the access token to make requests to the Shopify API.
8. Shopify returns the requested data.

https://shopify.dev/apps/auth/oauth#the-oauth-flow

We’re halfway there! We have an installable Shopify app that is a security nightmare, a GDPR liability, exclusively serves up a single static piece of text, and can’t interact with Shopify at all beyond the first stage of installation, but hey – it’s a Shopify app.

Status

Where we started in this post: We had an application server responding (albeit with garbage) to traffic originating from Shopify

Where we ended with this post: By tackling the first few steps of the Shopify OAuth flow, we have an INSTALLABLE Shopify application that can totally be installed in a store and then it’s in the list of installed apps. Also we have some handy looking data showing up at our callback URL

Next post: We complete the OAuth flow (glossing over a couple major TO-DOs) and obtain the credentials we need to make an authenticated request to Shopify on behalf of the store installing the application

Shopify app with Rust: Part 3 – the app server

In the second post (Connecting Shopify to your computer), I went over getting to the point where Shopify is sending web traffic from their store’s admin UI to your local machine. The devastating shortcoming of the project so far is that it literally does nothing except return a 502 Bad Gateway (and even that is thanks to ngrok), which isn’t a great user experience. The next step is to… build anything at all.

Setting the Shopify configuration aside, I’ve selected the Rocket web framework because… mostly it was a coin flip. I can’t say I’m particularly overjoyed in my experience using Rocket as it’s missing quite a few things I’m used to in other frameworks, but it’s Rust and there’s just not that much on offer yet. I have opted to use the 0.5 release candidate to use stable Rust. Notably, the main developer behind the framework has some not-volunteering-his-life-for-Internet-strangers going on which has hopefully only temporarily stalled Rocket.

Lets build the worst possible Rocket application and see what happens, so here’s some code:

# Cargo.toml

[package]
name = "shopify-template2"
version = "0.1.0"
edition = "2021"

[dependencies]
rocket = "0.5.0-rc.1"
// main.rs

fn rocket() -> rocket::Rocket<rocket::Build> {
	rocket::build()
}

#[rocket::main]
async fn main() {
	let _ = rocket().launch().await;
}

This is the reason for choosing 8000 as the port for ngrok, it’s the default used by Rocket. The the ol’ cargo run (ensuring that ngrok is still binding to 8000) will start a Rocket server. With the server running and ngrok forwarding traffing, going back into Shopify and trying to install the app in a test store again yields yet another new kind of failure – thrilling!

I see “rocket” in that 404, is that coming from _my_ machine?!

And in the terminal, we see the traffic is not just making it from Shopify -> ngrok -> our machine, it’s making it to the application server as well!

Now we’re getting somewhere!

Obviously it’d be surprising if Rocket worked out of the box with Shopify, so this failure is expected. Lets add an HTTP request handler so that we can start building out some functionality. At this point it does nothing but (hopefully) matches the URL Shopify is reaching out to, prints a line to the terminal, and redirects to… well, nowhere right now.

// src/main.rs

use rocket::{response::Redirect, routes, uri};

#[allow(unused_variables)]
#[rocket::get("/?<hmac>&<shop>&<timestamp>")]
pub fn install(hmac: &str, shop: &str, timestamp: &str) -> Redirect {
	println!("The route matches, that's progress...");
	Redirect::to(uri!("this-does-not-exist-yet"))
}

fn rocket() -> rocket::Rocket<rocket::Build> {
	rocket::build().mount("/", routes![install])
}

#[rocket::main]
async fn main() {
	let _ = rocket().launch().await;
}

Run the server again, try installing again, lets see what we get now…

Huge success! Installing the Shopify app has actually exercised some logic, a redirect to oblivion!

Confirmed by the server – traffic is coming in, a statement is being logged, a response is being issued. Significantly further than where we started, that’s a good stopping point.

One last step before we move on – for clarity’s sake going forward lets move the App URL from "/?<hmac>&<shop>&<timestamp>" to "/install?<hmac>&<shop>&<timestamp>". This will tidy things up very slightly in the future – remember to update your partner dashboard with the new App URL!

Update https://partners.shopify.com with /install!

Status

Where we started in this post: we had nothing but some Shopify accounts and some incoming web traffic, abandoned at our doorstep and ignored

Where we ended with this post: Installing the Shopify application executes {{logic}} on our local application server and returns a response, though not a very satisfying one

Next post: Implementing the first few steps of Shopify’s OAuth flow

Shopify app with Rust: Part 2 – Connecting Shopify to your computer

In the first post (Part 1: The preface), I went over the prerequisites for this Shopify application project. Now we’ll get to some of the more real stuff.

Step 1: Set up Shopify stuff

To do anything, you’ll need a partner account, and to test out our work you’ll need a development store. With both of those sorted, create a public app.

Fill in the required data

The “App URL” and at least one “Allowed redirection URL(s)” is required, lets use the defaults at this stage – “https://localhost/&#8221;. This won’t work, but we’ll find that out soon enough. Now we have a Shopify application! Lets try it out by installing it in our development store.

Test your app

Click through to your development store, aaaannnddd….

The app does not work

Ah, devastating failure right out of the gate. If you look at Shopify’s documented OAuth flow:

1. The merchant makes a request to install the app.
2. The app redirects to Shopify to load the OAuth grant screen and requests the required scopes.
3. Shopify displays a prompt to the merchant to give authorization to the app, and prompts the merchant to log in if required.
4. The merchant consents to the scopes and is redirected to the redirect_uri.
5. The app makes an access token request to Shopify including the client_id, client_secret, and code.
6. Shopify returns the access token and requested scopes.
7. The app uses the access token to make requests to the Shopify API.
8. Shopify returns the requested data.

https://shopify.dev/apps/auth/oauth#the-oauth-flow

We’re acting as the merchant here, so step 1 – check, we’re requesting to install the app. The next step, “the app redirects to Shopify” is definitely not happening. Not only is the app not redirecting, it’s not doing so hot on the whole “existing” front either. If you look at the page your browser tries to load, you’ll see something like:

https://localhost/?hmac=a4e2f622dec7da4ef778281da94d7ed41c3d6b627ec0498ac24ad5264bb02f6f&shop=toby-test-store2.myshopify.com&timestamp=1633291317

localhost, eh? https://localhost/ looks an awful lot like some corners we cut just moments ago. Navigating back into the app’s screen in the Shopify partner portal, we can open up the “App setup” page and revisit our choice of “App URL” to confirm our suspicions. Update it to a different value (make sure to save!), and try to install again

Same failure, but different

Confirmed – “App URL” is the URL Shopify is using to look for the app (imagine that). So now we just need a real URL… and I guess also an app that’s available at that URL.

Step 2: Obtain a public URL

If you have a VM in the cloud, there’s a solid chance you will either have a public IP associated with the VM or be able to purchase one. I tend to use Linode’s 5$ VMs, as they include a static IPv4 address (referral link). During development, something like ngrok is handy for instantly making your local PC accessible from the Internet. Going the ngrok route is pretty straightforward, so lets start there. We’ll use port 8000 for reasons that are vaguely relevant in the future. ngrok http 8000 is all that’s needed:

ngrok exposing port 8000

Would you look at that – ngrok is forwarding a public URL to the local machine. Now we have a public URL! It’s admittedly obscure, but for me (note this will be a dead or random link by the time you read this): https://2f51-204-237-50-186.ngrok.io

Let’s throw that into the Shopify app configuration as the App URL and see what happens. Update the URL, save the configuration, try installing it again aaaaaand:

Slightly different failure!

Now we’re getting somewhere! Traffic is seemingly showing up on our local machine, we’re just not doing anything with it. In ngrok you should similarly see evidence of some traffic:

Ooohhh, Bad Gateway

We have web traffic moving from Shopify -> ngrok -> our local machine, which is pretty solid progress. Unfortunately, nothing is happening with that traffic yet. This leads to a quite obvious next step – setting up a web server to handle this traffic!

Where we started in this post: we had nothing but a dream and a vague roadmap

Where we ended with this post: Installing the Shopify application drives web traffic to our local machine

Next post: Take the web traffic from a Shopify installation request and do anything with it

Shopify app with Rust: Part 1 – the preface

Why write this?

I’m trying to learn Rust, and building a Shopify application is an opportunity to do so. I find writing holds me accountable to actually knowing what I think I know, so selfishly it helps me enumerate everything involved in the foundation of the project.

Why read this?

Shopify’s docs are pretty extensive, but they lean very heavily on frameworks and tooling (Rails/single page applications) to the point of obscuring what’s actually required. I found trying to build an application that does not use these tools (vanilla JavaScript? Anything non-SPA-y?) frustrating, so I hope this helps provide some specific answers for other who want to stray from the prescribed tutorial.

Why skip this?

If you’re wanting the path of least resistance, you should use Shopify’s documentation and exactly the tools and technologies Shopify suggests in their documentation. Most of the community discussions and forum support is around these abstractions, so following the implementations in this post will lead you off on your own to an extent. It’s still eminently doable, but makes a better 2nd or 3rd app than a first app.

Also, a big caveat here is I’m just learning Rust. This will not be a tour of exemplary Rust code, it’ll be whatever I happen to be writing at the time and I certainly haven’t even mastered the fundamentals of the language.

With all that said, lets get to it!


Prerequisites

What do we actually need to do?

This list is actually reasonably long, but it should be mostly identical for every Shopify application.

  1. Create Shopify resources: https://shopify.dev/apps/auth/oauth#requirements. I won’t go into how to do this at all, read the Shopify documentation as it stands a better chance of being up to date.
    • a free developer account (partner account, in Shopify lingo) – you only need to pay the fee to be listed in the public app marketplace
    • a free development store, so you can test out your application
    • a free Shopify application
  2. Find somewhere publicly accessible to host the application (your machine works fine to start)
  3. Implement 2 OAuth APIs and associated validation
    1. app URL
    2. callback URL
    3. nonce validation
    4. HMAC validation
    5. JWT validation
  4. Implement a landing page that verifies we cannot access the page from the public internet and we can access it with a valid Shopify JWT
  5. Implement mandatory GDPR webooks: https://shopify.dev/apps/webhooks/mandatory

How are we going to do it?

By continuously failing in the most naive way possible until there’s nothing left to fail at! I go with this strategy because it’s typically how I learn things in a new environment. Essentially I’m going to do less than the bare minimum, discover what deficiencies exist, fix one, see if that gets me any further, then repeat. We’re slowly going to work down Shopify’s list of OAuth steps to start, then clean up the leftovers by implementing the stuff that nobody makes you do but you have to.

Step 0: Install requirements and set up Shopify stuff

I’m not going to copy/paste instructions on how to install Rust or SQLite as they’re both well documented – ensure you have that done, then continue.

To do anything, you’ll need a partner account, and to test out our work you’ll need a development store.

With the development environment requirements installed and Shopify accounts set up, you’re ready to go!

Shopify HMAC Verification with Rust

Every request or redirect from Shopify to your app’s server includes an hmac parameter that can be used to verify the authenticity of the request from Shopify.

https://shopify.dev/apps/auth/oauth#verification

I want to build a Shopify application, and I want to use it as an excuse to do some (likely painful) web development with Rust. I would like to learn Rust, but haven’t had many interesting projects to do it with. One of the things I tripped over when building a Shopify application was the very explicit guidance to verify the HMAC value.

Going into this, I had no idea what the HMAC value was, nor what I was supposed to do with it. The Shopify documentation is pretty explicit, but I was going to need even more of a hand holding, because I really didn’t have a clue. According to the docs:

  1. Get to a point where Shopify is sending a request to your application, e.g by installing your app in a Shopify store
  2. remove the hmac query string field from the query string
    • this includes the key, the value, the equal sign, and the delimiting & if present
  3. convert the hmac value from a string into a
  4. Compute the value of the resultant query string (with hmac removed) with a HMAC-SHA256 hash function, where the key is your Shopify Application’s API secret key
  5. compare (securely) the computed value with the provided value, discarding or otherwise not trusting the request if the comparison fails

The code

Disclaimers:

  • I have only used this for HTTP GET requests.
  • I used Rocket, having not used it before today
  • I’m new to Rust, so keep that in mind
// Cargo.toml
[package]
name = "ready-for-production-now"
version = "0.1.0"
edition = "2018"

[dependencies]
rocket = "0.5.0-rc.1"
regex = "1"
hmac = "0.11"
sha2 = "0.9"
hex = "0.4"
lazy_static = "1.4"
// main.rs
use hmac::{Hmac, Mac, NewMac};
use lazy_static::lazy_static;
use regex::Regex;
use rocket::http::Status;
use rocket::request::{FromRequest, Outcome, Request};
use sha2::Sha256;
use std::env;

type HmacSha256 = Hmac<Sha256>;

/// Type to integrate with Rocket's request guards
struct HmacParam();

/// The failure modes of HMAC verification, mostly for troubleshooting
#[derive(Debug)]
enum HmacParamError {
    NoQueryString,
    QueryStringButNoHmac,
    VerificationFailed,
}

lazy_static! {
    /// Note, this is not the best possible implementation. If the hmac value is the last in the query string, this
    /// will break due to a trailing '&' character. Maybe I'll fix that some day...
    static ref HMAC_REGEX: Regex = Regex::new(r"hmac=[^&]+&?").unwrap();
    /// I like to load in secrets with dotenv or similar, then consume them as environment variables
    static ref SHOPIFY_API_SECRET: String = env::var("SHOPIFY_API_SECRET").unwrap();
}

#[rocket::async_trait]
impl<'r> FromRequest<'r> for HmacParam {
    type Error = HmacParamError;

    async fn from_request(request: &'r Request<'_>) -> Outcome<Self, Self::Error> {
        // 1. Get the entire query string in one shot
        let query_string = match request.uri().query() {
            None => return Outcome::Failure((Status::Unauthorized, HmacParamError::NoQueryString)),
            Some(q) => q,
        };

        // 2. Grab the hmac query parameter (both key and value)
        let hmac = match query_string.segments().find(|&param| param.0 == "hmac") {
            None => {
                return Outcome::Failure((Status::Unauthorized, HmacParamError::QueryStringButNoHmac))
            }
            Some(h) => h,
        };

        // 3. Remove the hmac key and value from the query string, as they're excluded from the hash
        let query_string_without_hmac = HMAC_REGEX.replace(query_string.as_str(), "").into_owned();

        // 4. Create a HMAC SHA256 hash function using my app's API secret key as the hash key
        let mut hasher = HmacSha256::new_from_slice(&*SHOPIFY_API_SECRET.clone().into_bytes())
            .expect("HMAC key (Shopify API secret) was not appropriately set");

        // 5. Hash the remnants of the query string (with hmac removed
        hasher.update(&query_string_without_hmac.into_bytes());

        // 6. The value in the query string is the result of the hash function represented as hexadecimal, represented
        // as a string. I say that slighly weirdly because you can't just compare the value to the computed hash, the
        // string needs to be decoded as hexadecimal (e.g. the characters '02' are the numerical value 2). At that
        // point the resultant "number" can be compared with the output of the hash function
        let hmac = hex::decode(hmac.1).unwrap();

        // 7. Compare the Shopify-provided value to the freshly computed value
        match hasher.verify(&hmac) {
            Ok(_) => Outcome::Success(HmacParam {}),
            Err(_) => Outcome::Failure((Status::Unauthorized, HmacParamError::VerificationFailed)),
        }
    }
}

fn main() {
    // your code here
}
// example-usage.rs

// Notice hmac_guard: HmacParam! That's the request guard written above
#[allow(unused_variables)]
#[rocket::get("/install?<hmac>&<shop>&<timestamp>")]
fn auth_install(hmac: &str, shop: &str, timestamp: &str, _hmac_guard: HmacParam) -> Redirect {
	// this nonce should be kept to verify the next step
	let nonce = Uuid::new_v4(); 

	// I'm not being coy using constants
, you'll need to actually fill these out with values that make sense for your environment
	let redirect =format!("https://{shop}/admin/oauth/authorize?client_id={client_id}&scope={scopes}&redirect_uri={redirect_uri}&state={nonce}&grant_options[]={access_mode}", 
		shop = shop,
		client_id=*SHOPIFY_API_KEY,
		scopes=SCOPES,
		redirect_uri = *SHOPIFY_REDIRECT_ADDRESS,
		nonce=nonce,
		access_mode=ACCESS_MODE);

	Redirect::to(redirect)
}

What HMAC verification and why bother?

So, why even bother with this? I have not published an application to the Shopify marketplace, so what follows is a bit of speculation. From what I can tell with an in-development app, Shopify does not “test” to ensure you’re validating the HMAC. The Shopify app review guidelines also do not explicitly call out a requirement to do so lest you fail review, but that may be a thing once I get there.

From what I can tell, the intention of the HMAC verification is to ensure the integrity of the request both in its original authorship and in transit between Shopify and your application. As your API’s secret key is used as input to the hashing function, this guarantees that it is Shopify (or someone with access to your API’s secret key, at least) that has authored the request.

At the same time, any request body and all the headers (save the hmac value itself) are inputs to the hash, which guarantees that the content has not been modified/appended to in transit.

As someone with very little experience in this area, it’s unclear how frequently an HMAC verification check fails, and how often that would be because of foul play vs. software bugs, so I have no sense of how useful it is overall.

Edit: Originally I had this code returning Status::BadRequest, Shopify validates that the code returns a 401 Unauthorized (rather than a 400 Bad Request).

Unify Signal, WhatsApp and SMS in a personal Matrix server: Part 3 (WhatsApp)

Prerequisites

See Part 1 to get a running Synapse server based off a Docker-Compose file, example file included at the bottom of this post. Not necessary, but you can also check out Part 2 for a very similar post but regarding a Signal bridge instead of WhatsApp.

Matrix-WhatsApp Bridge

From a Matrix perspective, this is very similar to the Signal integration. The significant difference is that instead of deploying a daemon that receives the messages directly, this uses the WhatsApp web client. We’ll use the same initial configuration and registration setup, and we’ll also use a Postgres database to store the bridge state.

At this point, the Synapse server is running and stable, so we just have to tackle the bridge itself. Most of this content is based off what is available here: https://docs.mau.fi/bridges/go/whatsapp/index.html

  1. Update the docker-compose.yaml to contain an actual password for the whatsapp-bridge-db. If nothing else, this is a useful step to make sure you’re not mixing up which bridge is using which database
  2. Use an ephemeral Docker container to create the bridge’s configuration file, with the volume we’ll ultimately use in the Docker-Compose version:
$ docker run -it --rm -v whatsapp-bridge-data:/data dock.mau.dev/tulir/mautrix-whatsapp:latest
Didn't find a config file.
Copied default config file to /data/config.yaml
Modify that config file to your liking.
Start the container again after that to generate the registration file.
  1. Same as for Signal and Syanpse, we’ll update this configuration file:
    1. sudo nano $(docker inspect --format='{{.Mountpoint}}' whatsapp-bridge-data)/config.yaml
    2. Update a handful of things in this file:
      • under homeserver:
        • update address to be http://synapse:8008
        • update domain to be the domain you are ultimately going to use, for me it’s nuc.localdomain
      • under appservice:
        • update address to http://whatsapp-bridge:29328
        • under database update the type to be postgres and the uri to be the full connection string. For me that looks like: postgres://mautrix_whatsapp:ralGudPassword@whatsapp-bridge-db/whatsapp_bridge_db?sslmode=disable
          • Note: ensure you use ?sslmode=disable at the end of your connection string
      • under bridge
        • under permission add yourself as an administrator (e.g. "toby@nuc.localdomain": "admin")
    3. Ensure you save the file!
  2. use another ephemeral Docker container to consume the bridge’s configuration and create a registration file that can be used with Synapse
$ docker run -it --rm -v whatsapp-bridge-data:/data dock.mau.dev/tulir/mautrix-whatsapp:latest
Registration generated. Add the path to the registration to your Synapse config, restart it, then start the bridge.
Didn't find a registration file.
Generated one for you.
Copy that over to synapses app service directory.
  1. Now we need to get the registration.yaml for this bridge into Syanpse. Luckily, it’s in a volume that is mounted to both containers, so it’s straightforward. Edit the Syanpse server’s homeserver.yaml
    1. sudo nano $(docker inspect --format='{{.Mountpoint}}' synapse-data)/homeserver.yaml
    2. find app_service_config_files (if its your first bridge then it’s likely commented out, if so uncomment it)
    3. add the path to the the generated config.yaml
# A list of application service config files to use
app_service_config_files:
  - /whatsapp-bridge/registration.yaml
  1. Now you should be able to start the WhatsApp bridge and restart Synapse. Note that you might see a couple innocuous errors at the start as both are trying to start simultaneously and connect with one another.
  2. With the WhatsApp bridge running, it’s time to actually set up your WhatsApp account. This section is deployment agnostic, so you can follow the documentation here.
    1. Invite (or send a direct message, depending on your client) the WhatsApp bot. The bot will be named as you configured, with a default name of @whatsappbot, making mine for example: @whatsappbot:nuc.localdomain. You’ll get early feedback that it’ll work when you see the icon change to be WhatsApp’s.
Invite the WhatsApp bridge bot to chat
  1. send the message login to the bridge bot. Then in WhatsApp on your phone go to the Menu and select WhatsApp Web.
  2. Scan the QR code the bot has sent you to link the device.
Link your account

Now when you receive a WhatsApp message it should show up in the web client and be pushed to Matrix!


Example docker-compose.yaml

version: "3.7"

volumes:
  # We need to do some initial set up outside of Docker-Compose, so make these volumes external
  synapse-data:
    external: true
  whatsapp-bridge-data:
    external: true
  signal-bridge-data:
    external: true

  # The Signal daemon (Signald) and the Signal bridge (Mautrix-Signal) need to share a volume
  signald-data:

  # Use these named volumes just for convenience
  signal-bridge-db:
  synapse-db:
  whatsapp-bridge-db:

services:
  synapse-db:
    container_name: synapse-db
    image: postgres:13-alpine
    restart: unless-stopped
    environment:
      POSTGRES_USER: synapse
      POSTGRES_PASSWORD: <GENERATE A PASSWORD HERE> # Make sure to update this!
      POSTGRES_DB: synapse_db
      # ensure the database gets created correctly
      # https://github.com/matrix-org/synapse/blob/master/docs/postgres.md#set-up-database
      POSTGRES_INITDB_ARGS: --encoding=UTF-8 --lc-collate=C --lc-ctype=C
    volumes:
      - synapse-db:/var/lib/postgresql/data
    ports:
      - 5435:5432/tcp

  synapse:
    container_name: synapse
    image: docker.io/matrixdotorg/synapse:latest
    # Since synapse does not retry to connect to the database, restart upon failure
    restart: unless-stopped
    environment:
      - SYNAPSE_CONFIG_PATH=/data/homeserver.yaml
    volumes:
      - synapse-data:/data
      - signal-bridge-data:/signal-bridge
      - whatsapp-bridge-data:/whatsapp-bridge
    depends_on:
      - synapse-db
    ports:
      - 8008:8008/tcp
      - 8448:8448/tcp

  whatsapp-bridge-db:
    container_name: whatsapp-bridge-db
    image: postgres:13-alpine
    restart: unless-stopped
    environment:
      POSTGRES_USER: mautrix_whatsapp
      POSTGRES_DB: whatsapp_bridge_db
      POSTGRES_PASSWORD: <GENERATE A PASSWORD HERE> # Make sure to update this!
    volumes:
    - whatsapp-bridge-db:/var/lib/postgresql/data
    ports:
        - 5436:5432/tcp
  whatsapp-bridge:
    container_name: whatsapp-bridge
    image: dock.mau.dev/tulir/mautrix-whatsapp:latest
    restart: unless-stopped
    volumes:
    - whatsapp-bridge-data:/data
    ports:
      - 29318:29318/tcp
    depends_on:
      - whatsapp-bridge-db

  # Signal and bridge
  signald:
    container_name: signald
    image: finn/signald:latest
    restart: unless-stopped
    volumes:
    - signald-data:/signald
  signal-bridge-db:
    container_name: signal-bridge-db
    image: postgres:13-alpine
    restart: unless-stopped
    environment:
      POSTGRES_USER: mautrix_signal
      POSTGRES_DB: signal_bridge_db
      POSTGRES_PASSWORD: <GENERATE A PASSWORD HERE> # Make sure to update this!
    volumes:
    - signal-bridge-db:/var/lib/postgresql/data
    ports:
        - 5434:5432/tcp
  signal-bridge:
    container_name: signal-bridge
    image: dock.mau.dev/tulir/mautrix-signal
    restart: unless-stopped
    volumes:
    - signal-bridge-data:/data
    - signald-data:/signald
    ports:
      - 29328:29328/tcp
    depends_on:
      - signal-bridge-db
      - signald

“Windows” key doesn’t work on Advantage2 Keyboard

I very recently got a Kinesis Advantage2 keyboard. Needless to say, it’s different than the bog standard keyboard I was using before. My initial excitement was dampened slightly because when I plugged it into my desktop the Windows key (“super” key) didn’t work. Specifically, in Ubuntu it did not launch Dash.

Windows key! Or super key, if you prefer

A 40-page manual later:

great success!

Standard disclaimer about no guarantees this will work, but distilled down to the crucial steps:

  1. Hold the progm + shift + Esc and you should see the 4 lights in the middle flashing, then release the keys
  2. Hold the progm + F1 key to mount the keyboard as a USB drive
  3. Open the drive up, mine was called “ADVANTAGE2”
  4. Open the active directory
  5. Open the text file which has a name that corresponds to the layout you use, qwerty.txt for me
  6. Using the “Location tokens” (that’s what the manual calls them), remap the “Right Windows Button” to “Left Windows Button”. The file should look like:
  1. Save the file
  2. Eject the volume
  3. take the Windows key for a test drive!

Unify Signal, WhatsApp and SMS in a personal Matrix server: Part 2 (Signal)

Prerequisites

See Part 1 to get a running Synapse server based off a Docker-Compose file, example file included at the bottom of this post

Matrix-Signal Bridge

Alright, assuming you’re up to date with part 1 then a Synapse server is running and accessible. Huge first step! Now to integrate a Signal bridge. This is going to be very similar to the WhatsApp bridge set up that is upcoming in part 3.

  1. Start a Signal daemon like Signald
    • docker-compose up -d signald
  2. update the docker-compose.yaml to contain an actual password for signal-bridge-db
  3. use an ephemeral Docker container to create the bridge’s configuration file
$ docker run -it --rm -v signal-bridge-data:/data dock.mau.dev/tulir/mautrix-signal:latest
Didn't find a config file.
Copied default config file to /data/config.yaml
Modify that config file to your liking.
Start the container again after that to generate the registration file.
  1. Same as for Synapse, we’ll update this configuration file
    1. sudo nano $(docker inspect --format='{{.Mountpoint}}' signal-bridge-data)/config.yaml
    2. Update a whole bunch of stuff in here:
      • under homeserver
        • update address to be http://synapse:8008
        • update domain to be the domain you are ultimately going to use, for me it’s nuc.localdomain
      • under appservice
        • update address to http://signal-bridge:29328
        • update database to the full Postgres database string, e.g. postgres://mautrix_signal:reallyStrongPassword!2@signal-bridge-db/signal_bridge_db
      • under signal
        • update socket_path to /signald/signald.sock
        • update outgoing_attachment_dir to /signald/attachments
        • update avatar_dir to /signald/avatars
        • update data_dir to /signald/data
      • under bridge
        • under permission add yourself as an administrator (e.g. "toby@nuc.localdomain": "admin")
    3. ensure you save the file!
  2. use another ephemeral Docker container to consume the bridge’s configuration and create a registration file that can be used with Synapse
$ docker run -it --rm -v signal-bridge-data:/data dock.mau.dev/tulir/mautrix-signal:latest
Registration generated and saved to /data/registration.yaml
  1. Now we need to get the registration.yaml for this bridge into Syanpse. Luckily, it’s in a volume that is mounted to both containers, so it’s straightforward. Edit the homeserver.yaml
    • sudo nano $(docker inspect --format='{{.Mountpoint}}' synapse-data)/homeserver.yaml
    • find app_service_config_files (it’s likely commented out). Uncomment it and add the path to the the generated config.yaml
# A list of application service config files to use
#
app_service_config_files:
  - /signal-bridge/registration.yaml
  1. Now you should be able to start both the Signal bridge and Synapse. Note that it may error out a couple times as both are trying to start up talking to one another.
  2. Now is time to actually set up your Signal account with the Signal daemon. These instructions are basically from here.
    1. Invite (or send a direct message, depending on your client) the Signal bot which will be based on your own server name: (@signalbot:nuc.localdomain). You’ll get early feedback that it’ll work when you see the icon change to be Signal’s
  1. send the message link to the bridge bot, then in Signal go to the Settings then Linked devices and Add
  2. Scan the QR code, and hopefully you should be on your way!

At this point, when you receive a Signal message it should show upon the daemon and get pushed to Matrix. If this the first message you’ve received from that contact, you’ll be invited to a new chat (or new group, if it’s a group message) and then be able to chat away! A logical next double-puppeting, which makes the whole experience nicer. I have yet to find a way to proactively crawl existing conversations so they’re present in Matrix right off the bat.

Example docker-compose.yaml

version: "3.7"

volumes:
  # We need to do some initial set up outside of Docker-Compose, so make these volumes external
  synapse-data:
    external: true
  whatsapp-bridge-data:
    external: true
  signal-bridge-data:
    external: true

  # The Signal daemon (Signald) and the Signal bridge (Mautrix-Signal) need to share a volume
  signald-data:

  # Use these named volumes just for convenience
  signal-bridge-db:
  synapse-db:

services:
  synapse-db:
    container_name: synapse-db
    image: postgres:13-alpine
    restart: unless-stopped
    environment:
      POSTGRES_USER: synapse
      POSTGRES_PASSWORD: <GENERATE A PASSWORD HERE> # Make sure to update this!
      POSTGRES_DB: synapse_db
      # ensure the database gets created correctly
      # https://github.com/matrix-org/synapse/blob/master/docs/postgres.md#set-up-database
      POSTGRES_INITDB_ARGS: --encoding=UTF-8 --lc-collate=C --lc-ctype=C
    volumes:
      - synapse-db:/var/lib/postgresql/data
    ports:
      - 5435:5432/tcp

  synapse:
    container_name: synapse
    image: docker.io/matrixdotorg/synapse:latest
    # Since synapse does not retry to connect to the database, restart upon failure
    restart: unless-stopped
    environment:
      - SYNAPSE_CONFIG_PATH=/data/homeserver.yaml
    volumes:
      - synapse-data:/data
      - signal-bridge-data:/signal-bridge
      - whatsapp-bridge-data:/whatsapp-bridge
    depends_on:
      - synapse-db
    ports:
      - 8008:8008/tcp
      - 8448:8448/tcp

  whatsapp-bridge:
    container_name: whatsapp-bridge
    image: dock.mau.dev/tulir/mautrix-whatsapp:latest
    restart: unless-stopped
    volumes:
    - whatsapp-bridge-data:/data
    ports:
      - 29318:29318/tcp

  # Signal and bridge
  signald:
    container_name: signald
    image: finn/signald:latest
    restart: unless-stopped
    volumes:
    - signald-data:/signald
  signal-bridge-db:
    container_name: signal-bridge-db
    image: postgres:13-alpine
    restart: unless-stopped
    environment:
      POSTGRES_USER: mautrix_signal
      POSTGRES_DB: signal_bridge_db
      POSTGRES_PASSWORD: <GENERATE A PASSWORD HERE> # Make sure to update this!
    volumes:
    - signal-bridge-db:/var/lib/postgresql/data
    ports:
        - 5434:5432/tcp
  signal-bridge:
    container_name: signal-bridge
    image: dock.mau.dev/tulir/mautrix-signal
    restart: unless-stopped
    volumes:
    - signal-bridge-data:/data
    - signald-data:/signald
    ports:
      - 29328:29328/tcp
    depends_on:
      - signal-bridge-db
      - signald