A neat thing that Strava premium has is a personal heatmap. It can look something like this:
Strava Global Heatmap
Garmin Connect does not seem to offer a personal heat map at all, only global heat maps. I want my own personal heat map for a number of reasons, but mostly because they’re cool. So… how do I go about that? First, looking for prior art I found this post: https://www.musings.ch/2020/02/24/all-garmin-activities-on-a-map/ which was pretty encouraging that it was at least possible.
Convert activity-containing FIT files to GPX files
I used the fitparser crate to read the data, geo-types to represent each FIT point in an intermediate representation, then the gpx crate to write those to a new GPX file
Plot GPX files on the map with the map provided via Leaflet.js and the GPX drawing provided via leaflet-gpx
Et voilà:
“Heat map”
There are some notable differences from what I originally hoped for with this first pass. The start/end points are rendered as pins, and the “heat” aspect is purely GPS imprecision. Still, I’m thrilled – it’s super cool and I love it. I can also see everywhere I’ve traveled and recorded an activity as well.
The previous post (get OAuth access token) covered retrieving the OAuth access token that can be used for persistent permission to retrieve a shop’s data. To make use of that in the context of an application, we’ll need to attach it to a request. There are a lot of small supporting pieces here, so you’ll need a bit of patience.
First things first, if you’re actually coding along then when you open the application within Shopify’s dashboard, you’re likely seeing something disappointingly like:
Firefox can’t open this page
This is a security feature that is enabled by Rocket by default that prevents our application from being embedded within another webpage (e.g. the Shopify partner dashboard). Specifically, this is the Rocket shield. There are still some pieces of Rocket that are unimplemented, e.g. support for Content-Security-Policy, so let’s disable some protection entirely to make it embeddable!
With the X-Frame-Options header removed (via Frame), the page should now be embeddable within the Shopify portal (not that we have much to show…).
Making a really good and not at all terrible UI
Aligned with the theme of this series, lets just barrel forwards naively and fix stuff when it doesn’t work. Any self respecting application needs some form of UI, so far we’re lacking that regard. Rather than add that logic in our /install or /callback request handlers, lets add a new API. As these blog posts are not about UI technologies, we’re going to do about the laziest possible thing and return static HTML directly with no supporting tooling. We’ll spice it up just a tiny bit by making it parameterizable with the shop’s name:
// main.rs
fn build_index_html(shop: &str) -> String {
format!(
r#"<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>R'al cool Shopify app</title>
<script>
function doFancyFrontEndThings() {{
window.alert("Ya dun clicked the button");
}}
</script>
</head>
<body>
<h1>Welcome to this Shopify Application, {}</h1>
<button onclick="doFancyFrontEndThings()">Click me!</button>
</body>
</html>"#,
shop,
)
}
Add a new route that returns this page so that we can see our really rockin’ UI:
Now if we visit the app again we should actually be able to see HTML, which is dramatically cooler than the terribleness we had previously:
Glorious rendered HTML!
Feels like we’re getting close… an installable application with a UI?! The big piece that’s missing is getting Shopify data.
Authorized request to Shopify
Currently we have the access token for our test shop being printed out into the application server logs (bad practice!), so lets use it. Run the application and save the access token somewhere, we’ll hardcode it shortly. Shopify has a number of different APIs and a mix off REST based and GraphQL based interfaces. For some extra complexity, there’s not feature parity between the two. In general, it seems the GraphQL version is intended to be the “future”, so we’ll go down that route. Most APIs relevant for simple apps will come from the Admin API section.
If you’re trying to craft your own query, the Shopify GraphiQL explorer is a great tool to use. If you were doing this for real, you’d likely use a GraphQL client in Rust (e.g. graphql_client), but to keep our dependencies as minimal as possible we’re going to cut some corners. We will add the serde_json crate though, as a halfway between unstructured text and full deserialized structs.
As we’re not trying to tackle “how to work with Shopify’s API in general”, lets use retrieving the shop’s name as placeholder logic. To do so, we’ll need to form a GraqhQL request and submit it to the right endpoint. Well need 4 pieces to build this HTTP POST:
the endpoint: https://{shop}.myshopify.com/admin/api/2021-07/graphql.json
the Content-Type header: Content-Type: application/graphql
the shop’s access token as a header: X-Shopify-Access-Token: {token}
The actual GraphQL query
As with the HTML, make a constant for the very simple GraphQL query. In a real app instead of getting the shop’s name you might get recent orders or customers:
If we traverse this JSON object with wild abandon, we can update our UI so it’s outrageously personalized:
// main.rs
...
let response_body: serde_json::Value = response
.json()
.await
.map_err(|_| "Error getting response body as json")?;
let shop_name = &response_body["data"]["shop"]["name"].as_str().unwrap();
Ok(content::Html(build_index_html(shop_name)))
}
...
Whoa – data from the API in our UI!
Status
Where we started in this post: We had an installable application that would solicit an OAuth access token, but not do anything with it
Where we ended with this post: We tied up a couple loose ends, built a dynamite UI, made a request with the OAuth access token, and served up a UI with data retrieved from Shopify. Things are looking particularly legit around these parts.
Next post: We’ll avoid hard coding the very secret OAuth access token in the browser and store it in memory in Rocket instead
The last post (starting OAuth) tackled the first pass at the app installation API that serves Shopify’s OAuth flow. It also covered declaring the redirect URI, so Shopify sends the application server a web request, the server responds with a redirect to Shopify to authorize the installation, and then Shopify sends the user back to the application server.
A quick refresher on Shopify’s described OAuth flow:
1. The merchant makes a request to install the app. 2. The app redirects to Shopify to load the OAuth grant screen and requests the required scopes. 3. Shopify displays a prompt to the merchant to give authorization to the app, and prompts the merchant to log in if required. 4. The merchant consents to the scopes and is redirected to the redirect_uri. 5. The app makes an access token request to Shopify including the client_id, client_secret, and code. 6. Shopify returns the access token and requested scopes. 7. The app uses the access token to make requests to the Shopify API. 8. Shopify returns the requested data.
We have an authorized request knocking at the door of the application server (/callback API) but we’re not doing anything with it. In this post we’re going to tackle the body of:
The goal here is to address: “The app makes an access token request to Shopify including the client_id, client_secret, and code.” If you take a close look at the output of the previous attempt to install an application, it seems we’ve missed quite a few parameters on the callback API:
Callback API invocation
Specifically on:
GET /callback?code=8a99f98c7deb52660f76219dee5ada50&hmac=6cb4c00bf66110d41cb4d7e26c59ade80d3b322de7ca0f28e54ac9b8dace4f4b&host=dG9ieS10ZXN0LXN0b3JlMi5teXNob3BpZnkuY29tL2FkbWlu&shop=toby-test-store2.myshopify.com&state=random-value×tamp=1639244833 text/html
There are 6 query parameters!
code: The authorization code provided in the redirect, this is what we were after in the previous step and will use in the next step to get a token!
looks like: 8a99f98c7deb52660f76219dee5ada50
hmac: This is a signature we’ll use later to verify the request is unaltered from Shopify, it’s a bit ol’ TO-DO right now though
The timestamp, state, and hmac we can ignore entirely for the time being (this is a development app, we can’t ignore these in production). The host is useful, but not immediately, so let’s ignore it for the time being as well. The code and shop are exactly what we need for the next step. First, let’s make them accessible in our request handler:
Now, what are we ultimately doing with these values? We have a code that represents the Shopify store’s agreement to install the application with the requested permissions. Now that we have that code, we’re going back to Shopify to say “Look, the shop agreed – can we access their data?”. This happens by exchanging the authorization code for an access token. The access token will let us make requests to Shopify that are authorized to access the granting store’s data. Before we dive into that too much, lets actually get an access token.
We need the application server to make a request. To do so, we’ll need an HTTP client. Looking at lib.rs, reqsounds godwest seems like it may be overkill but will likely do the job. We’re using virtually no features of the HTTP client library, and in this case it has to be synchronous regardless of whether the underlying code is sync or async.
Add reqwest to your Cargo.toml:
[package]
name = "template"
version = "0.1.0"
edition = "2021"
[dependencies]
rocket = "0.5.0-rc.1"
reqwest = { version = "0.11" }
Now let’s send an HTTP POST to https://{shop}/admin/oauth/access_token. In the spirit of just doing stuff, let’s give it a shot and see what happens. We’ll ignore Reqwest’s advice to share a client for the time being (another TO-DO?) and go for it. Lets also take the opportunity to change the return type to a Result just because it makes the code a bit easier to work with.
// main.rs
#[rocket::get("/callback?<code>&<shop>")]
pub async fn callback(code: &str, shop: &str) -> Result<&'static str, &'static str> {
println!("The code is: {}", code);
println!("The shop is: {}", shop);
let access_token_uri = format!("https://{}/admin/oauth/access_token", shop);
let client = reqwest::Client::new();
let response = client
.post(access_token_uri)
.send()
.await
.map_err(|_| "Error sending request")?;
println!("Response status: {}", response.status());
let response_body = response
.text()
.await
.map_err(|_| "Error getting response body as text")?;
println!("{}", response_body);
Ok("Now what...")
}
Some good things: the request itself succeeds in sending and a response comes back. Areas for improvement, the request yields an HTTP 400 Bad Request and the body is HTML. If you render the HTML, it looks something like this:
HTTP 400 Bad Request
As it says on Shopify, this is a JSON body with 3 properties:
client_id: this is our application’s API key, coming from the Shopify partner dashboard for the app
client_secret: this is our applications API secret key, also coming from the Shopify partner dashboard for the app
code: this is the code query parameter from the callback URL we have is populated
So, lets grab the values and try again! client_id and client_secret come from the Shopify partner dashboard:
This is where API key and API Secret key are coming from
The code is a query parameter in the original request. We’re crafting a JSON request, and basically as soon as anyone mentions JSON it’s time to add the Serde crate. While we’re there, we’ll enable the json feature of Reqwest so we can use Serde:
[package]
name = "template"
version = "0.1.0"
edition = "2021"
[dependencies]
rocket = "0.5.0-rc.1"
reqwest = { version = "0.11", features = ["json"] }
serde = "1"
One of the easiest ways I’ve found to craft a JSON request is to define a type for the request body and then let Serde serialize it to JSON. Adding the appropriate import and a stuct definition:
Now we can use that and amend our request so it contains the correct body:
#[rocket::get("/callback?<code>&<shop>")]
pub async fn callback(code: &str, shop: &str) -> Result<&'static str, &'static str> {
let access_token_uri = format!("https://{}/admin/oauth/access_token", shop);
let body = AccessTokenRequest {
client_id: "801f137828580f71e33ad6b14f75e533",
client_secret: "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",
code,
};
let client = reqwest::Client::new();
let response = client
.post(access_token_uri)
.json(&body)
.send()
.await
.map_err(|_| "Error sending request")?;
println!("Response status: {}", response.status());
let response_body = response
.text()
.await
.map_err(|_| "Error getting response body as text")?;
println!("{}", response_body);
Ok("Now what...")
}
Start up the application server, try installing or opening the app and:
Voilà! An access token!
We have an access token for the shop! As a side note, this is effectively a password that can be used to access a store’s data. Never reveal the value, and be diligent to not let it show up in screenshots in a blog post or similar.
As a quality of life improvement, let’s also deserialize the response body so it’s easier to use. As a quirk of the deserialization, we can only use owned types, so we need String instead of slices:
Once again, referring back to the Shopify OAuth flow we’ve now tackled two more steps:
1. The merchant makes a request to install the app. 2. The app redirects to Shopify to load the OAuth grant screen and requests the required scopes. 3. Shopify displays a prompt to the merchant to give authorization to the app, and prompts the merchant to log in if required. 4. The merchant consents to the scopes and is redirected to the redirect_uri. 5. The app makes an access token request to Shopify including the client_id, client_secret, and code. 6. Shopify returns the access token and requested scopes. 7. The app uses the access token to make requests to the Shopify API. 8. Shopify returns the requested data.
Where we started in this post: We had an app that “installable” and we could both request and be granted permissions for a store installing the app. Unfortunately we could not leverage those permissions to do anything
Where we ended with this post: We’ve exchanged the ephemeral authorization a store grants us for a permanent authorization. In practice this looks like using the provided code value for an OAuth access token. With this token, we can (theoretically at this point) make requests to Shopify for data that belongs to the associated store
Next post: Make an authorized request and obtain a shop’s data!
Previously (the app server) we got to the point where a Shopify store owner could try to install an application, they would make it to our server, our server would do some work, and then the user would get sent to oblivion. Now we’re going to try and correct that, send the user somewhere slightly more useful.
For the time being, we’re going to completely ignore both the HMAC validation and the nonce. They’re important, but for the sake of satisfaction checking things off they can be deferred (can’t go to production without it though). The next step is to show the application permissions prompt. This is pretty straightforward, we just need to collect a handful of values and then redirect the Shopify application:
shop: the name of the shop installing the app, looks something like shop-name.myshopify.com
API key: the API key of the application (not the API secret key), available in the partner portal – looks something like c2ecfe6c6b3745cfaaa9a539b1dfe8a7
scopes: a comma separated list of all of the permissions the app will have. A possible value might be something like: read_products,read_product_listings,read_orders,read_customers
redirect URI: where you want the user to go after everything is said and done, e.g. the landing page for your application
nonce: a random value that can be checked at the next step to ensure the integrity of the process – can be anything, e.g. a UUID works great
access mode: the lifetime of the the token you’ll ultimately get that will allow you to do the things specified in scopes on the user’s behalf. This has to be either per-user or value
Collecting all the values takes a bit, but once you’ve done so the next step is very straightforward. You jam them into a a new URL and this is ultimately what we redirect to:
Implementing this is as straightforward as it seems – amend the previously defined route to redirect somewhere useful:
#[allow(unused_variables)]
#[rocket::get("/install?<hmac>&<shop>&<timestamp>")]
pub fn install(hmac: &str, shop: &str, timestamp: &str) -> Redirect {
let api_key = "801f137828580f71e33ad6b14f75e533"; // Obviously don't use _MY_ key, swap this out for your own
let scopes = "read_products,read_product_listings,read_orders,read_customers"; // These will be custom for you depending on your goals
let nonce = "random-value"; // TODO: This should be unique to each OAuth flow, so hold your nose for a bit...
let redirect = format!("https://{shop}/admin/oauth/authorize?client_id={client_id}&scope={scopes}&redirect_uri={redirect_uri}&state={nonce}&grant_options[]={access_mode}",
shop = shop,
client_id = api_key,
scopes = scopes,
redirect_uri = "this-does-not-exist-yet", // Redirecting to oblivion
nonce = nonce,
access_mode = "value");
Redirect::to(redirect)
}
Now, try the install dance again and see what we get!
“The redirect_uri is not whitelisted”
Now isn’t that interesting, we redirected somewhere but something else is going on! We’re one step further and have uncovered a brand new way to screw this up. We redirected the user, but in the very beginning we just filled in the field in our app’s configuration with nonsense. Time to go back and fill in a value that lines up with the app. I used “this-does-not-exist-yet” in the code above, so it’s appropriate to be skeptical this will solve all our problems, but lets see.
In the partner portal, click on your app and then “App setup”, “find Allowed redirection URL(s)” and add the value we’re redirecting to (“this-does-not-exist-yet”), then save.
Fails validation
Looks like the value we’ve used is too fake to be useful! Update the redirect_uri in the code to something that looks more like a URL (e.g. https://this-does-not-exist-yet.test) and put the corresponding value in the configured “Allowed redirection URL(s)”.
let redirect = format!("https://{shop}/admin/oauth/authorize?client_id={client_id}&scope={scopes}&redirect_uri={redirect_uri}&state={nonce}&grant_options[]={access_mode}",
shop = shop,
client_id = api_key,
scopes = scopes,
redirect_uri = "https://this-does-not-exist-yet.test", // <-- This is what changed here
nonce = nonce,
access_mode = "value");
Now restart your server and do the installation dance again:
Made it to the permissions page
We’ve snuck by the validation! We’re so close to having an installable Shopify application, just click that sweet, sweet “Install unlisted app” button and…
We’ve redirected to oblivion again
I have to be honest, I saw this one coming. Obviously the value we used for redirect_uri was a bit less useless. We’ll need our application to have at least one more route, so we can redirect users there at this point. Lets fix this quick:
add a brain-dead API, e.g. I chose (/callback) that will return some static text (just so we have some response)
mount the API
update the redirect_uri in your code so to reflect this new value
re-re-update the “Allowed redirection URL(s)” in your Shopify partner portal’s app configuration to be whatever your real value is for the API we just introduce
For me, the whole URL for this new callback API looks like: https://2f51-204-237-50-186.ngrok.io/callback. Note that because this is using ngrok, every time you re-establish an ngrok connection you’ll have to update this value and keep it in sync between Shopify configuration and your application code, lest you run into the errors we saw above. This means it’s easiest to keep the ngrok application running the whole time you’re developing so you don’t have to go whitelist the address in your “Allowed Redirection URL(s)” field in your Shopify app configuration constantly.
Shockingly enough, at this point when we try again we will technically have installed your application into the store. To test again you can delete it if you’d like, but we’ll end up the same place if you try and reinstall it in the same store, so dealer’s choice.
After making the code and configuration changes, restart the server, reinstall the application, annnnnddd…..
Successful redirection!
Looking at the terminal:
terminal output
We can see a request to: /callback?code=94872a044567b4711e544497b7ce2afc&hmac=14dd56d06cd2962b13cc833dc2472173eeba0223135cdd0b99f5fc45039755ee&host=dG9ieS10ZXN0LXN0b3JlMi5teXNob3BpZnkuY29tL2FkbWlu&shop=toby-test-store2.myshopify.com&state=random-value×tamp=1633314364
That’s a lot of stuff that we’re currently not doing anything with. Revisiting Shopify’s described OAuth flow:
1. The merchant makes a request to install the app. 2. The app redirects to Shopify to load the OAuth grant screen and requests the required scopes. 3. Shopify displays a prompt to the merchant to give authorization to the app, and prompts the merchant to log in if required. 4. The merchant consents to the scopes and is redirected to the redirect_uri. 5. The app makes an access token request to Shopify including the client_id, client_secret, and code. 6. Shopify returns the access token and requested scopes. 7. The app uses the access token to make requests to the Shopify API. 8. Shopify returns the requested data.
We’re halfway there! We have an installable Shopify app that is a security nightmare, a GDPR liability, exclusively serves up a single static piece of text, and can’t interact with Shopify at all beyond the first stage of installation, but hey – it’s a Shopify app.
Status
Where we started in this post: We had an application server responding (albeit with garbage) to traffic originating from Shopify
Where we ended with this post: By tackling the first few steps of the Shopify OAuth flow, we have an INSTALLABLE Shopify application that can totally be installed in a store and then it’s in the list of installed apps. Also we have some handy looking data showing up at our callback URL
Next post: We complete the OAuth flow (glossing over a couple major TO-DOs) and obtain the credentials we need to make an authenticated request to Shopify on behalf of the store installing the application
In the second post (Connecting Shopify to your computer), I went over getting to the point where Shopify is sending web traffic from their store’s admin UI to your local machine. The devastating shortcoming of the project so far is that it literally does nothing except return a 502 Bad Gateway (and even that is thanks to ngrok), which isn’t a great user experience. The next step is to… build anything at all.
Setting the Shopify configuration aside, I’ve selected the Rocket web framework because… mostly it was a coin flip. I can’t say I’m particularly overjoyed in my experience using Rocket as it’s missing quite a few things I’m used to in other frameworks, but it’s Rust and there’s just not that much on offer yet. I have opted to use the 0.5 release candidate to use stable Rust. Notably, the main developer behind the framework has some not-volunteering-his-life-for-Internet-strangers going on which has hopefully only temporarily stalled Rocket.
Lets build the worst possible Rocket application and see what happens, so here’s some code:
# Cargo.toml
[package]
name = "shopify-template2"
version = "0.1.0"
edition = "2021"
[dependencies]
rocket = "0.5.0-rc.1"
This is the reason for choosing 8000 as the port for ngrok, it’s the default used by Rocket. The the ol’ cargo run (ensuring that ngrok is still binding to 8000) will start a Rocket server. With the server running and ngrok forwarding traffing, going back into Shopify and trying to install the app in a test store again yields yet another new kind of failure – thrilling!
I see “rocket” in that 404, is that coming from _my_ machine?!
And in the terminal, we see the traffic is not just making it from Shopify -> ngrok -> our machine, it’s making it to the application server as well!
Now we’re getting somewhere!
Obviously it’d be surprising if Rocket worked out of the box with Shopify, so this failure is expected. Lets add an HTTP request handler so that we can start building out some functionality. At this point it does nothing but (hopefully) matches the URL Shopify is reaching out to, prints a line to the terminal, and redirects to… well, nowhere right now.
Run the server again, try installing again, lets see what we get now…
Huge success! Installing the Shopify app has actually exercised some logic, a redirect to oblivion!
Confirmed by the server – traffic is coming in, a statement is being logged, a response is being issued. Significantly further than where we started, that’s a good stopping point.
One last step before we move on – for clarity’s sake going forward lets move the App URL from "/?<hmac>&<shop>&<timestamp>" to "/install?<hmac>&<shop>&<timestamp>". This will tidy things up very slightly in the future – remember to update your partner dashboard with the new App URL!
Where we started in this post: we had nothing but some Shopify accounts and some incoming web traffic, abandoned at our doorstep and ignored
Where we ended with this post: Installing the Shopify application executes {{logic}} on our local application server and returns a response, though not a very satisfying one
Next post: Implementing the first few steps of Shopify’s OAuth flow
In the first post (Part 1: The preface), I went over the prerequisites for this Shopify application project. Now we’ll get to some of the more real stuff.
The “App URL” and at least one “Allowed redirection URL(s)” is required, lets use the defaults at this stage – “https://localhost/”. This won’t work, but we’ll find that out soon enough. Now we have a Shopify application! Lets try it out by installing it in our development store.
Test your app
Click through to your development store, aaaannnddd….
1. The merchant makes a request to install the app. 2. The app redirects to Shopify to load the OAuth grant screen and requests the required scopes. 3. Shopify displays a prompt to the merchant to give authorization to the app, and prompts the merchant to log in if required. 4. The merchant consents to the scopes and is redirected to the redirect_uri. 5. The app makes an access token request to Shopify including the client_id, client_secret, and code. 6. Shopify returns the access token and requested scopes. 7. The app uses the access token to make requests to the Shopify API. 8. Shopify returns the requested data.
We’re acting as the merchant here, so step 1 – check, we’re requesting to install the app. The next step, “the app redirects to Shopify” is definitely not happening. Not only is the app not redirecting, it’s not doing so hot on the whole “existing” front either. If you look at the page your browser tries to load, you’ll see something like:
localhost, eh? https://localhost/ looks an awful lot like some corners we cut just moments ago. Navigating back into the app’s screen in the Shopify partner portal, we can open up the “App setup” page and revisit our choice of “App URL” to confirm our suspicions. Update it to a different value (make sure to save!), and try to install again
Same failure, but different
Confirmed – “App URL” is the URL Shopify is using to look for the app (imagine that). So now we just need a real URL… and I guess also an app that’s available at that URL.
Step 2: Obtain a public URL
If you have a VM in the cloud, there’s a solid chance you will either have a public IP associated with the VM or be able to purchase one. I tend to use Linode’s 5$ VMs, as they include a static IPv4 address (referral link). During development, something like ngrok is handy for instantly making your local PC accessible from the Internet. Going the ngrok route is pretty straightforward, so lets start there. We’ll use port 8000 for reasons that are vaguely relevant in the future. ngrok http 8000 is all that’s needed:
ngrok exposing port 8000
Would you look at that – ngrok is forwarding a public URL to the local machine. Now we have a public URL! It’s admittedly obscure, but for me (note this will be a dead or random link by the time you read this): https://2f51-204-237-50-186.ngrok.io
Let’s throw that into the Shopify app configuration as the App URL and see what happens. Update the URL, save the configuration, try installing it again aaaaaand:
Slightly different failure!
Now we’re getting somewhere! Traffic is seemingly showing up on our local machine, we’re just not doing anything with it. In ngrok you should similarly see evidence of some traffic:
Ooohhh, Bad Gateway
We have web traffic moving from Shopify -> ngrok -> our local machine, which is pretty solid progress. Unfortunately, nothing is happening with that traffic yet. This leads to a quite obvious next step – setting up a web server to handle this traffic!
Where we started in this post: we had nothing but a dream and a vague roadmap
Where we ended with this post: Installing the Shopify application drives web traffic to our local machine
Next post: Take the web traffic from a Shopify installation request and do anything with it
I’m trying to learn Rust, and building a Shopify application is an opportunity to do so. I find writing holds me accountable to actually knowing what I think I know, so selfishly it helps me enumerate everything involved in the foundation of the project.
Why read this?
Shopify’s docs are pretty extensive, but they lean very heavily on frameworks and tooling (Rails/single page applications) to the point of obscuring what’s actually required. I found trying to build an application that does not use these tools (vanilla JavaScript? Anything non-SPA-y?) frustrating, so I hope this helps provide some specific answers for other who want to stray from the prescribed tutorial.
Why skip this?
If you’re wanting the path of least resistance, you should use Shopify’s documentation and exactly the tools and technologies Shopify suggests in their documentation. Most of the community discussions and forum support is around these abstractions, so following the implementations in this post will lead you off on your own to an extent. It’s still eminently doable, but makes a better 2nd or 3rd app than a first app.
Also, a big caveat here is I’m just learning Rust. This will not be a tour of exemplary Rust code, it’ll be whatever I happen to be writing at the time and I certainly haven’t even mastered the fundamentals of the language.
No real reason to use SQLite here over anything else, just easy to set up
A publicly accessible host, directly or via something like ngrok: https://ngrok.com/
What do we actually need to do?
This list is actually reasonably long, but it should be mostly identical for every Shopify application.
Create Shopify resources: https://shopify.dev/apps/auth/oauth#requirements. I won’t go into how to do this at all, read the Shopify documentation as it stands a better chance of being up to date.
a free developer account (partner account, in Shopify lingo) – you only need to pay the fee to be listed in the public app marketplace
a free development store, so you can test out your application
a free Shopify application
Find somewhere publicly accessible to host the application (your machine works fine to start)
Implement 2 OAuth APIs and associated validation
app URL
callback URL
nonce validation
HMAC validation
JWT validation
Implement a landing page that verifies we cannot access the page from the public internet and we can access it with a valid Shopify JWT
By continuously failing in the most naive way possible until there’s nothing left to fail at! I go with this strategy because it’s typically how I learn things in a new environment. Essentially I’m going to do less than the bare minimum, discover what deficiencies exist, fix one, see if that gets me any further, then repeat. We’re slowly going to work down Shopify’s list of OAuth steps to start, then clean up the leftovers by implementing the stuff that nobody makes you do but you have to.
Step 0: Install requirements and set up Shopify stuff
I’m not going to copy/paste instructions on how to install Rust or SQLite as they’re both well documented – ensure you have that done, then continue.
Every request or redirect from Shopify to your app’s server includes an hmac parameter that can be used to verify the authenticity of the request from Shopify.
I want to build a Shopify application, and I want to use it as an excuse to do some (likely painful) web development with Rust. I would like to learn Rust, but haven’t had many interesting projects to do it with. One of the things I tripped over when building a Shopify application was the very explicit guidance to verify the HMAC value.
Going into this, I had no idea what the HMAC value was, nor what I was supposed to do with it. The Shopify documentation is pretty explicit, but I was going to need even more of a hand holding, because I really didn’t have a clue. According to the docs:
Get to a point where Shopify is sending a request to your application, e.g by installing your app in a Shopify store
remove the hmac query string field from the query string
this includes the key, the value, the equal sign, and the delimiting & if present
convert the hmac value from a string into a
Compute the value of the resultant query string (with hmac removed) with a HMAC-SHA256 hash function, where the key is your Shopify Application’s API secret key
compare (securely) the computed value with the provided value, discarding or otherwise not trusting the request if the comparison fails
// main.rs
use hmac::{Hmac, Mac, NewMac};
use lazy_static::lazy_static;
use regex::Regex;
use rocket::http::Status;
use rocket::request::{FromRequest, Outcome, Request};
use sha2::Sha256;
use std::env;
type HmacSha256 = Hmac<Sha256>;
/// Type to integrate with Rocket's request guards
struct HmacParam();
/// The failure modes of HMAC verification, mostly for troubleshooting
#[derive(Debug)]
enum HmacParamError {
NoQueryString,
QueryStringButNoHmac,
VerificationFailed,
}
lazy_static! {
/// Note, this is not the best possible implementation. If the hmac value is the last in the query string, this
/// will break due to a trailing '&' character. Maybe I'll fix that some day...
static ref HMAC_REGEX: Regex = Regex::new(r"hmac=[^&]+&?").unwrap();
/// I like to load in secrets with dotenv or similar, then consume them as environment variables
static ref SHOPIFY_API_SECRET: String = env::var("SHOPIFY_API_SECRET").unwrap();
}
#[rocket::async_trait]
impl<'r> FromRequest<'r> for HmacParam {
type Error = HmacParamError;
async fn from_request(request: &'r Request<'_>) -> Outcome<Self, Self::Error> {
// 1. Get the entire query string in one shot
let query_string = match request.uri().query() {
None => return Outcome::Failure((Status::Unauthorized, HmacParamError::NoQueryString)),
Some(q) => q,
};
// 2. Grab the hmac query parameter (both key and value)
let hmac = match query_string.segments().find(|¶m| param.0 == "hmac") {
None => {
return Outcome::Failure((Status::Unauthorized, HmacParamError::QueryStringButNoHmac))
}
Some(h) => h,
};
// 3. Remove the hmac key and value from the query string, as they're excluded from the hash
let query_string_without_hmac = HMAC_REGEX.replace(query_string.as_str(), "").into_owned();
// 4. Create a HMAC SHA256 hash function using my app's API secret key as the hash key
let mut hasher = HmacSha256::new_from_slice(&*SHOPIFY_API_SECRET.clone().into_bytes())
.expect("HMAC key (Shopify API secret) was not appropriately set");
// 5. Hash the remnants of the query string (with hmac removed
hasher.update(&query_string_without_hmac.into_bytes());
// 6. The value in the query string is the result of the hash function represented as hexadecimal, represented
// as a string. I say that slighly weirdly because you can't just compare the value to the computed hash, the
// string needs to be decoded as hexadecimal (e.g. the characters '02' are the numerical value 2). At that
// point the resultant "number" can be compared with the output of the hash function
let hmac = hex::decode(hmac.1).unwrap();
// 7. Compare the Shopify-provided value to the freshly computed value
match hasher.verify(&hmac) {
Ok(_) => Outcome::Success(HmacParam {}),
Err(_) => Outcome::Failure((Status::Unauthorized, HmacParamError::VerificationFailed)),
}
}
}
fn main() {
// your code here
}
// example-usage.rs
// Notice hmac_guard: HmacParam! That's the request guard written above
#[allow(unused_variables)]
#[rocket::get("/install?<hmac>&<shop>&<timestamp>")]
fn auth_install(hmac: &str, shop: &str, timestamp: &str, _hmac_guard: HmacParam) -> Redirect {
// this nonce should be kept to verify the next step
let nonce = Uuid::new_v4();
// I'm not being coy using constants
, you'll need to actually fill these out with values that make sense for your environment
let redirect =format!("https://{shop}/admin/oauth/authorize?client_id={client_id}&scope={scopes}&redirect_uri={redirect_uri}&state={nonce}&grant_options[]={access_mode}",
shop = shop,
client_id=*SHOPIFY_API_KEY,
scopes=SCOPES,
redirect_uri = *SHOPIFY_REDIRECT_ADDRESS,
nonce=nonce,
access_mode=ACCESS_MODE);
Redirect::to(redirect)
}
What HMAC verification and why bother?
So, why even bother with this? I have not published an application to the Shopify marketplace, so what follows is a bit of speculation. From what I can tell with an in-development app, Shopify does not “test” to ensure you’re validating the HMAC. The Shopify app review guidelines also do not explicitly call out a requirement to do so lest you fail review, but that may be a thing once I get there.
From what I can tell, the intention of the HMAC verification is to ensure the integrity of the request both in its original authorship and in transit between Shopify and your application. As your API’s secret key is used as input to the hashing function, this guarantees that it is Shopify (or someone with access to your API’s secret key, at least) that has authored the request.
At the same time, any request body and all the headers (save the hmac value itself) are inputs to the hash, which guarantees that the content has not been modified/appended to in transit.
As someone with very little experience in this area, it’s unclear how frequently an HMAC verification check fails, and how often that would be because of foul play vs. software bugs, so I have no sense of how useful it is overall.
Edit: Originally I had this code returning Status::BadRequest, Shopify validates that the code returns a 401 Unauthorized (rather than a 400 Bad Request).
See Part 1 to get a running Synapse server based off a Docker-Compose file, example file included at the bottom of this post. Not necessary, but you can also check out Part 2 for a very similar post but regarding a Signal bridge instead of WhatsApp.
Matrix-WhatsApp Bridge
From a Matrix perspective, this is very similar to the Signal integration. The significant difference is that instead of deploying a daemon that receives the messages directly, this uses the WhatsApp web client. We’ll use the same initial configuration and registration setup, and we’ll also use a Postgres database to store the bridge state.
At this point, the Synapse server is running and stable, so we just have to tackle the bridge itself. Most of this content is based off what is available here: https://docs.mau.fi/bridges/go/whatsapp/index.html
Update the docker-compose.yaml to contain an actual password for the whatsapp-bridge-db. If nothing else, this is a useful step to make sure you’re not mixing up which bridge is using which database
Use an ephemeral Docker container to create the bridge’s configuration file, with the volume we’ll ultimately use in the Docker-Compose version:
$ docker run -it --rm -v whatsapp-bridge-data:/data dock.mau.dev/tulir/mautrix-whatsapp:latest
Didn't find a config file.
Copied default config file to /data/config.yaml
Modify that config file to your liking.
Start the container again after that to generate the registration file.
Same as for Signal and Syanpse, we’ll update this configuration file:
under database update the type to be postgres and the uri to be the full connection string. For me that looks like: postgres://mautrix_whatsapp:ralGudPassword@whatsapp-bridge-db/whatsapp_bridge_db?sslmode=disable
Note: ensure you use ?sslmode=disable at the end of your connection string
under bridge
under permission add yourself as an administrator (e.g. "toby@nuc.localdomain": "admin")
Ensure you save the file!
use another ephemeral Docker container to consume the bridge’s configuration and create a registration file that can be used with Synapse
$ docker run -it --rm -v whatsapp-bridge-data:/data dock.mau.dev/tulir/mautrix-whatsapp:latest
Registration generated. Add the path to the registration to your Synapse config, restart it, then start the bridge.
Didn't find a registration file.
Generated one for you.
Copy that over to synapses app service directory.
Now we need to get the registration.yaml for this bridge into Syanpse. Luckily, it’s in a volume that is mounted to both containers, so it’s straightforward. Edit the Syanpse server’s homeserver.yaml
find app_service_config_files (if its your first bridge then it’s likely commented out, if so uncomment it)
add the path to the the generated config.yaml
# A list of application service config files to use
app_service_config_files:
- /whatsapp-bridge/registration.yaml
Now you should be able to start the WhatsApp bridge and restart Synapse. Note that you might see a couple innocuous errors at the start as both are trying to start simultaneously and connect with one another.
With the WhatsApp bridge running, it’s time to actually set up your WhatsApp account. This section is deployment agnostic, so you can follow the documentation here.
Invite (or send a direct message, depending on your client) the WhatsApp bot. The bot will be named as you configured, with a default name of @whatsappbot, making mine for example: @whatsappbot:nuc.localdomain. You’ll get early feedback that it’ll work when you see the icon change to be WhatsApp’s.
Invite the WhatsApp bridge bot to chat
send the message login to the bridge bot. Then in WhatsApp on your phone go to the Menu and select WhatsApp Web.
Scan the QR code the bot has sent you to link the device.
Link your account
Now when you receive a WhatsApp message it should show up in the web client and be pushed to Matrix!
Example docker-compose.yaml
version: "3.7"
volumes:
# We need to do some initial set up outside of Docker-Compose, so make these volumes external
synapse-data:
external: true
whatsapp-bridge-data:
external: true
signal-bridge-data:
external: true
# The Signal daemon (Signald) and the Signal bridge (Mautrix-Signal) need to share a volume
signald-data:
# Use these named volumes just for convenience
signal-bridge-db:
synapse-db:
whatsapp-bridge-db:
services:
synapse-db:
container_name: synapse-db
image: postgres:13-alpine
restart: unless-stopped
environment:
POSTGRES_USER: synapse
POSTGRES_PASSWORD: <GENERATE A PASSWORD HERE> # Make sure to update this!
POSTGRES_DB: synapse_db
# ensure the database gets created correctly
# https://github.com/matrix-org/synapse/blob/master/docs/postgres.md#set-up-database
POSTGRES_INITDB_ARGS: --encoding=UTF-8 --lc-collate=C --lc-ctype=C
volumes:
- synapse-db:/var/lib/postgresql/data
ports:
- 5435:5432/tcp
synapse:
container_name: synapse
image: docker.io/matrixdotorg/synapse:latest
# Since synapse does not retry to connect to the database, restart upon failure
restart: unless-stopped
environment:
- SYNAPSE_CONFIG_PATH=/data/homeserver.yaml
volumes:
- synapse-data:/data
- signal-bridge-data:/signal-bridge
- whatsapp-bridge-data:/whatsapp-bridge
depends_on:
- synapse-db
ports:
- 8008:8008/tcp
- 8448:8448/tcp
whatsapp-bridge-db:
container_name: whatsapp-bridge-db
image: postgres:13-alpine
restart: unless-stopped
environment:
POSTGRES_USER: mautrix_whatsapp
POSTGRES_DB: whatsapp_bridge_db
POSTGRES_PASSWORD: <GENERATE A PASSWORD HERE> # Make sure to update this!
volumes:
- whatsapp-bridge-db:/var/lib/postgresql/data
ports:
- 5436:5432/tcp
whatsapp-bridge:
container_name: whatsapp-bridge
image: dock.mau.dev/tulir/mautrix-whatsapp:latest
restart: unless-stopped
volumes:
- whatsapp-bridge-data:/data
ports:
- 29318:29318/tcp
depends_on:
- whatsapp-bridge-db
# Signal and bridge
signald:
container_name: signald
image: finn/signald:latest
restart: unless-stopped
volumes:
- signald-data:/signald
signal-bridge-db:
container_name: signal-bridge-db
image: postgres:13-alpine
restart: unless-stopped
environment:
POSTGRES_USER: mautrix_signal
POSTGRES_DB: signal_bridge_db
POSTGRES_PASSWORD: <GENERATE A PASSWORD HERE> # Make sure to update this!
volumes:
- signal-bridge-db:/var/lib/postgresql/data
ports:
- 5434:5432/tcp
signal-bridge:
container_name: signal-bridge
image: dock.mau.dev/tulir/mautrix-signal
restart: unless-stopped
volumes:
- signal-bridge-data:/data
- signald-data:/signald
ports:
- 29328:29328/tcp
depends_on:
- signal-bridge-db
- signald
I very recently got a Kinesis Advantage2 keyboard. Needless to say, it’s different than the bog standard keyboard I was using before. My initial excitement was dampened slightly because when I plugged it into my desktop the Windows key (“super” key) didn’t work. Specifically, in Ubuntu it did not launch Dash.
Technically Rural 