Recognizing ArcGIS

In order to use this technique, you need to be sure that the data you're looking for is being hosted by ArcGIS. The easiest way to confirm this is by looking at the URL - it'll often contain the word arcgis:

The other easy way to recognize ArcGIS is by the look-and-feel of their dashboards:

or their browser icons:

Once you have a good feeling that you're looking at an ArcGIS dashboard, it's time to go looking for the underlying data.

Accessing the Underlying Data

ArcGIS is API-driven, so our task here is to find a way to interact with the API and retrieve the data in the form we want. Because of the many ways ArcGIS dashboards can be set up, I've found the most reliable way to find the API is using Chrome's DevTools to see where the dashboard's API requests are sent.

With DevTools open, browse to the ArcGIS page you'd like to find the data for, and look at the Network tab, typing  query into the Filter filed. You'll see something like this:

Those query?f=json... lines are the API requests we're trying to find. Right-click on one of them, select Copy → Copy as link address, and then paste that into your browser's location bar. In my example, I get https://services8.arcgis.com/
aCyQID5qQcyrJMm2/arcgis/rest/services/RHA_CurrentStats2_Public2
/FeatureServer/0/queryf=json&where=RHAShort%3D'Central_RHA'
&returnGeometry=false&spatialRel=esriSpatialRelIntersects&outFields
=*&resultOffset=0&resultRecordCount=25&resultType=standard&cacheHint=true. Delete everything after /rest/services/, and then go to that page - in my example, I end up with https://services8.arcgis.com/aCyQID5qQcyrJMm2/arcgis/rest/services/. This is a list of all the different ArcGIS data sources that have been made public on this server, and looks something like this:

On this list, it's important to be selective about the data you pick. You're looking at the raw data here, without the context of how it's captured, how up-to-date it is, how it's been processed, etc. You'll often see several version of the "same" data (e.g. Covid_19_PHC_Cases_Current and Covid_19_PHC_Cases_Current_v2) and not know how or why they're different,

Once you've found a service that sounds interesting, click into it, and then click into the Layer that has the data you want:

On this screen, you can see the fields in each record (Fields) and when the data was last updated (Last Edit Date). Click Query at the bottom (next to Supported Operations), and you'll get the API query page:

You have three fields to update before pressing Query (GET) at the bottom:

1. Set Where to 1=1. This tells the system that you want all records where 1=1, which is another way of saying you want all of them.
2. Set Out fields to *. This tells the system that you want all fields in each record. Once you understand what's there, you can change this to be a comma-separated list of only the fields you want.
3. Set Return Geometry to False, assuming you're looking for data values and not mapping polygons.

Once you've done that and press Query (GET), you'll see the first page of your data displayed. From that data, you can select the fields you're interested in, and put them into the Out fields box. When you're ready to go, change the Format box to JSON, and press Query (GET) again.

This time, you'll end up with a JSON-only result that looks something like this:

and a URL that looks like https://services8.arcgis.com/aCyQID5qQcyrJMm2
/ArcGIS/rest/services/Covid_19_PHC_Cases_Current_v2/FeatureServer
/0/query?where=1%....

This page is your raw data in machine-readable form, and that URL pulls new data each time you visit it. You can now add this into your data pipeline, and use the identical data that's driving the Government visualizations. You can also do interesting things like watching how the data changes over time (by downloading it every day), like I've done here.

Alexa, what's the heat set to?
- Sorry, what device?
Kitchen
- That's not supported yet

Alexa, what's the temperature set to?
- The heat's set to twenty
Alexa, set the heat to nineteen
- Sorry, what device?
Kitchen
- It's set to heat

Alexa, set the temperature to nineteen
- The heat's set to nineteen

I'll call it natural-language when I don't have to remember which synonym turns up the heat.

If you've received one of these, don't panic. Your computer hasn't been hacked, but you should change the passwords you care about - especially if you use the same password in several places.

What happened?

Every now and then you might see a news article about a big data breach, where someone broke into a company's computer system and got a list of usernames and passwords. These lists are then sold to extortionists and scammers, for them to use in scams like this one (or identity theft).

Bulk emails are cheap to send, so a scammer could run a profit on a scam like this if only one or two people in the world pay the "ransom". This is why they don't care that the email looks amateur and why they don't bother really customizing it for you: these scams are designed to be as low-effort as possible for the scammer, and by having it look suspicious, it filters out people who are less likely to fall for the scam .

To see where the password probably came from, put your email into haveibeenpwned.com. This is a free service run by a researcher who gets copies of most of the data breaches out there and indexes them. You'll see that many breaches are a few years old, which explains why the password in the email may be one you stopped using.

What do I do?

Most importantly, don't send any money to the address in the email. There is no recording , and there's no one that has all your contacts to send it to.

The key thing is to now realize that a lot of bad people on the internet have the combination of email and password that were included in that email, and they're trying to use them to get into all sorts of accounts you have (taking advantage of the fact that most people use the same password in a few places). This is called "credential stuffing", and is a popular type of hacking.

The most important thing you can do now is change your passwords. Change any service that you know used the password included in the email, and change your critical passwords (email, bank, social media) even if they didn't. Change each one to a different and unrelated password, and don't use words or simple substitutions like switching 0 for O or 1 for l.

Once that's done, check haveibeenpwned.com to see what other passwords of yours are in the wild. Change the password on any service you see listed there.

Since you're changing all your passwords, this is a great time to start using a password manager like 1Password or LastPass. Password managers are secure apps that store all your passwords, and keep track of things like when you last changed them or if they've been found in a data breach.

The idea is that you only have to remember one (complex) password for the password manager, and then the app keeps track of all the individual passwords that it creates for every service. They have browser plugins and mobile apps too, so that you never have to type a password in again. Using a password manager and letting it generate a unique password for every site is the best thing you can do for your online security, and it makes your life easier as a bonus.

Someone has bought this expensive piece of property, and then they just hand you the keys for it, cash your cheques, and do maintenance - or at least, that's the best case.

Often, though, the best case doesn't happen. Things break and they don't get fixed, your landlord does things they're not supposed to, or you and they have a difference of agreement on something. When that happens, the best plan is to try and fairly work something out informally. When that doesn't work, the Ontario legal system has a special place to deal with the situation: the Landlord and Tenant Board.

In my case, my landlord and I disagreed on who was responsible for fixing some broken cabinets. We tried to work it out, but after a lot of back-and-forth we couldn't - so I applied for an order to force them to fix my cabinets, and they applied for an order to force me to pay for fixing their cabinets.

In the end, we reached a mediated settlement that both of us were happy with. While preparing for that, though, I found there wasn't much out there about the experience from the perspective of a self-represented tenant.

Here's what I learned.

Structure of the Board

The Landlord and Tenant Board (LTB) is a tribunal created by the Rental Tenancies Act, charged with resolving disputes between landlords and tenants. This means that while it's not a court like on Law & Order, its judgements are real and legally binding.

Hearings are conducted by Members, who are career civil servants. Members may or may not have had legal training before they joined the LTB, but by the time they handle your case they'll know the Rental Tenancies Act inside and out, and will be experienced in how to run a session. Like judges, they can issue binding decisions and order fines. Unlike judges, they don't deal with criminal or civil matters, and they can't send you to jail or have you arrested.

You start an interaction with the LTB by filing a notice or application, which is a form asking the Board to order someone to do (or not do) something - as examples, this could be a tenant asking the Board to order their landlord to stop entering their unit, or a landlord asking the Board to order their tenant to move out. In either case, the person that filed the application is known as the applicant, and person receiving the order is the respondent.

You don't need a lawyer to appear before the LTB; especially in simpler cases, both sides representing themselves is pretty common. You can hire a lawyer, though, and if one side has one there's no obligation for the other side to have one. You can also hire a paralegal, bring a friend, or have a friend/family member who has legal training help you out (although there are some restrictions on this if you don't pay them). The important thing is that everyone involved only has one role, meaning that your friend who's sitting at the table and helping you out can't also be called as a witness.

Preparation

Preparation for a hearing consists of three main areas: understanding precedents and the law, creation of your narrative, and the collection & presentation of documents.

Precedents & the Law

In order to make a decent case, you need to know the law - so read it. At the LTB, the main governing laws are the Residential Tenancies Act and the by-laws of the community where the property is. Both of these are pretty straightforward as laws go, and can be read in an afternoon. You should also read the LTB's rules, guides and interpretations, which are like cheat-sheets for how the LTB usually handles certain parts of the law. The Advocacy Centre for Tenants Ontario also releases great tip-sheets on specific situations.

Once you've found the relevant part of the law, you'll likely see that your situation is slightly different than the exact one covered by the text. This is OK and normal - it doesn't mean that you've fallen into some hole in the legal system. Rather, it means that your task becomes to convince the Member that while there may be a small difference, your situation is what the law is talking about, and that you should get the remedy the law says.

If there's something in the law you're really not sure of, you can call a Community Legal Clinic. They're overworked and underpaid, but can likely point you in the right direction. Some communities may have tenants' rights groups with helplines, like FMTA (for tenants in Toronto).

From watching Law & Order, you may think that the most important argument you have is based on precedents - which isn't really the case at the LTB. Since the Board isn't a court, Members don't have to follow precedents directly, and they can get offended if you keep throwing references at them. Two or three references in one place is usually fine, but beyond that you're tempting fate. It's also important to present them not as "here's a binding thing you have to follow", but rather as a very respectful "here's how some other Member thought about this situation".

Reading old decisions is great, though, for seeing how your issue has been handled in the past. Searching CanLII for keywords from the part of the law you're interested in can help you find relevant LTB decisions. I found them most useful for suggesting arguments which won't work, and for showing how the average Member will deal with definitions and terms which aren't covered in the text of the law.

At this point, you should have a pretty good idea of what laws apply to your situation, what remedy those laws entitle you to, and how a few cases around them have been handled in the past. Armed with that, you're now ready to start crafting the story you'll use to explain why your interpretation is the right one.

Narrative

Your narrative is the story of what happened and led you to be at the LTB, as you see it. This is what you'll explain at the start of your hearing, and what you'll use to position and contextualize all your responses and questions. This should be informed by your understanding of the law, and will influence how you select and organize your evidence.

The important thing here is that while you should be using facts and descriptions in a way that supports your story, the facts still have to be true (from your perspective). As an example, let's say the issue is who's responsible for fixing a wall which collapsed. The landlord's narrative will likely be "the tenant did X, which caused the wall to fall down". In that case, your narrative could be "I don't know why the wall fell down, but it wasn't my fault". Both of those narratives reflect the truth as seen by each party, despite them being different.

If there are obvious things that happened which don't benefit your story, you should still include them (as the landlord certainly will). This shows to the Member that you're not trying to pull a fast one on them, and gives you the opportunity to put them into the context that they happened in, and mitigate their impact.

Lastly, while your narrative should be personal and told to promote the outcome you want, you shouldn't make it emotional or centered around a sob-story. Members see that every day; when you keep emotion out of it, you're showing that you're there for business, and want to be judged on your merits - not ignored as someone who's hoping the law will magically fix their problems.

When you set up your narrative properly, the rest of the process is about selling that narrative to the Member. If they believe yours is what happened, you win - if not, you don't.

Documents

The Board places a lot of emphasis on communication and timing, so the minute you think something may end up at the Board you should start planning for this. That means trying to keep all discussion between you and the landlord over email, and if you have to have a phone call or in-person meeting, immediately sending them a follow-up email that recounts your version of what was discussed (so you have a contemporary record of it). The goal here is to "transform" the conversation into an email so that you can refer to it easier and enter it as evidence. Ideally the landlord agrees with your read-out of the call, but even if not, it's much better than nothing.

You'll also want to prepare a summary spreadsheet of all communications, showing who initiated it, a one-line summary of what was said, and a time. The goal is to show that you're the reasonable and proactive one, and it's the landlord who's not being helpful.

Similarly, it's useful to make a timeline of important events in the process. Include things like when you moved in, when the incident took place, when any important conversations took place, and when any LTB filings happened. This will be helpful when you're debating the exact order things took place in, which can be critical.

Lastly, don't forget any photos or other relevant documents you want to refer to. Things like copies of your lease, articles about your landlord, and repair quotes. Even if they'll also be introduced by the landlord, putting them in your package means you can present them in your context, and use them to tell your narrative. If you've sent them to the landlord in advance - which you should - that's good to mention as well.

Take all these documents, create a simple cover page and table of contents with tabs, and get them bound at a copy shop. Make it look like something you'd give your CEO. This is now the package you'll give to the landlord and Member at the start of your hearing. If you can make most of this package before your hearing and mail it in to the LTB, you'll be in even better shape - as the Member will have read all your evidence without hearing the landlord's side.

The Day of the Hearing

My application was heard at the LTB's Eastern Office, in Ottawa. It's a non-descript floor of an office building. I arrived about 45 minutes before my hearing was scheduled to start, and there were a few others milling around. Nothing happened until it was 30 minutes before the scheduled start, which was when my notice had told me to show up.

At this point, three things happened:

1. A Commissionaire came out and started taking attendance, having everyone present sign in. You're tracked by your application number, so make sure you have that handy (and in the case where you're there for several applications, you need to sign in for each separately)
2. The organizer of the tenant duty counsel came out and reminded everyone that there was free legal advice available for tenants
3. The organizer of the mediation service came out and reminded everyone that mediation is available, should both sides desire it

If you elect to both consult the duty counsel and do mediation, don't worry about scheduling - the staff will arrange things so that your mediation spot doesn't happen until after you're done with duty counsel, and your hearing doesn't start until after you're done with mediation. If they miss it and your application gets called early, tell the Member right off the bat and they'll reorder it to give you some time.

My first stop was to sign up for a duty counsel slot - I did this even before they made the announcement, as I saw the sign-up list on a door.

Tenant Duty Counsel

Duty counsel is a service provided by Legal Aid. In my case, it was a pair of local law students who worked out of the community legal clinic. They did a quick interview to get the facts of my case and a list of specific questions I had, and then went off to speak with their supervising lawyer.

After ten minutes they came back with some general guidance, answers to the specific questions I had asked, and three important new pieces of information:

• If you're asking for a rent abatement because of loss of enjoyment or use of your unit, you should file a Form T2 (Application about Tenant Rights) in addition the original form where you ask for the abatement. The content may be the same, but the requests are handled under different standards depending on the form used.
• The usual amount given for a rent abatement is 10%-20%. More is possible, but you need to show very strong evidence for it.
• If you're asking for compensation for damage for personal belongings, the Board doesn't use strict replacement value - they depreciate the value based on how long you've had the item in question. In my case, this means that I wouldn't be likely to get much for the damage to my table as it's 15+ years old.

After I was done with duty counsel, I went to the hearing room to watch some proceedings and get a sense of how things work. On my way in, the mediation organizer stopped me and asked if I'd be interested in mediation. I said yes, and she said that she'd arrange the schedule so we had some time to chat before the application was called.

The Hearing

From speaking to others, I gather this varies a lot between Members adjudicating the hearing. The Member on my day was conversational, but very stern.

After giving both sides a preemptive lecture about how she doesn't take well to interruptions, the Member would ask the applicant to give their side of the story. Every time a document was mentioned, she would ensure that both she and the respondent were given a copy. After a few minutes she'd ask clarifying questions - all-in, each side would get about eight minutes to say their piece. This not a place for grand speeches; the Member was looking for the facts and the narrative around them, and nothing emotional or extra.

Following the applicant, the respondent would be given the chance to tell their side, with the Member asking similar questions - but this time, also to the applicant when the respondent said something that didn't match. Imagine Judge Judy, but taking place in a generic boardroom. Same self-represented feeling, same issues, same type of person leading the hearing while also questioning both sides.

While this is going on, people are coming and going from the hearing room, and the Commissionaire, mediation organizer, and duty counsel organizers are grabbing people. About halfway through the first contested application, it was my time for mediation.

Mediation

Mediation is a free and optional service offered by the LTB to help get the sides to a solution without an actual hearing. It's you, the landlord, and a neutral third-party (who works for the Board) chatting about your application. The mediator will ask questions, let both sides present their perspectives, and try to facilitate the discussion. It's a very informal discussion, with no rules of procedure.

The mediator will have read the application before starting the session, but it'll likely still start with both sides telling their narratives. From there, how it goes depends on the mediator and the positions of both sides.

In the best case, there's an agreement on how to settle the application. The mediator writes up an agreement that both sides sign, and you're done for the day. That agreement isn't an order of the Board, but it is legally binding - so if one side violates it, it's an easier process to come back to and get an order.

In cases where there isn't an overall agreement, the goal of the mediator is to get agreement on smaller things to make the actual hearing more streamlined. In this case the mediator will write out a letter the agreed points and give it to both sides and the Member, and it'll be what opens up the hearing.

If you do mediation, your goal should be to get the other side to make the arguments that they'll make in the hearing, and to see any evidence they'll be presenting. This gives you a bit more time to think about how you'll present your story, and preemptively rebut the positions they'll use. You can also change your approach if it seems like it isn't working, as the mediator isn't allowed to tell the Member about any of the discussions that took place.

In my case, mediation was successful. My mediator reminded both sides about precedents and rules, answered a few questions about what could be changed on the applications, offered perspective on which requests may not fly with the specific Member holding the hearings that day, and prodded appropriately when discussions were breaking down. I walked out of the LTB office with a signed mediation agreement that I was happy with, and was pretty close to what I could have expected to get at a hearing.

Tips

• Always file a Form T2 (Application about Tenant Rights). At worst, it'll be rejected by the Member - but if not, it gives you a second chance to make your argument about how the landlord's action is disrupting your "reasonable enjoyment of the rental unit".
• If all your applications are about the same core issue (or if the landlord has also filed something about your issue), see if you can get them all heard together. It'll be less of an inconvenience for you, and hearing all the arguments in one sitting makes it easier to tell the your larger story.
• You should address the Member "Mr./Ms. [last name]" or "Sir/Ma'am". They're not judges, so no need for "Your Honour" or anything more formal. Most people there will be dressed in business casual, including the lawyers & paralegals.
• Take advantage of low expectations. Most tenants who show up at the LTB are emotional and disorganized, so that's who most landlords and Members expect to deal with. If you show up as organized, rational, and polite-but-firm, you'll send a message to the landlord that you're there for business, and they they can't just make things up and walk all over you.

As you do your investigation and prototyping to help decide if the cost is worth it, I suggest starting with the following question:

Am I happy with my service discovery system?

In many ways, service discovery is at the heart of Envoy. If a product is going to create a mesh between containers and act as a smart communication bus between them, it needs to be able to find all those containers and address them directly.

Directly, here, is critical. Envoy needs something - an API or a DNS server - which can return the IPs of all containers^[1] of a particular service. In the simplest case, this could even be a configuration file in each Envoy container, so long as you have a way to keep it up-to-date. The IP of a load-balancer isn't sufficient here, as the whole point of Envoy is that it does its own load-balancing across long-running HTTP/2 connections.

If you don't have this, there's a good chance that Envoy is more than you need. For many use-cases, putting containers behind load-balancers and then distributing those load-balancer addresses is a great solution.

Really?

As an example of why service discovery is so core, let's consider how to deploy a new container. In a traditional architecture, we^[2] bring a new container online by making an API call to the load-balancer, waiting for health-checks to pass, and making a second API call to drain connections and then remove the old container.

In the world of Envoy - where routing is powered by service discovery - sending traffic to a new container is a side-effect of launching it. Once the new container is running it'll show up in service discovery, which will cause Envoy to start health-checking it, which will result in it getting traffic. Similarly, terminating the old container will cause it to fall out of service discovery and fail health checks, either of which will cause it to no longer get traffic.

I'm convinced

Even if you have a great service discovery system in place, Envoy may not be for you. It's a very new piece of technology that was written with some specific problems in mind, and there's a good chance that you don't have those problems. If you think you do, however, it's got some great backers and shows huge promise.

If you don't have a service discovery system in place that you're happy with, you're probably best served by continuing with what you have, or starting with something simpler. When your system gets complex enough that you've had some other reason to get good at service discovery, then it might be time to take another look at Envoy.

Zero-downtime deploys - the ability to release a new version of your code to production without taking the site down - are a key component of continuous delivery.

In the early days of AWS, this was a pain to do. We had auto-scaling groups (ASGs) that didn't play that well with ELBs, and you often ended up having to build tooling to do what seemed like a simple operation: "Update the instances in this ASG to use the launch configuration it now has". In theory, this changed with the release of the UpdatePolicy attribute on an ASG. From the blog post:

Today’s new feature allows you to perform a rolling deployment of an Auto Scaling Group within a CloudFormation stack. Instead of updating all of the instances in a group at the same time, you can now replace or modify the instances in a step-by-step fashion...This feature will increase availability of your application during an update.

The part before the ellipsis - updating instances in a step-by-step fashion - works quite well. The part after - increasing availability - doesn't, unless you use this in a very particular way. Before explaining why, I'll step back a second and explain how I expected this to work.

Let's assume I'm building a feature to increase availability during a deploy. My building blocks are:

1. A load-balancer with a robust health-check mechanism
2. A scaling system which knows how to use the health-check of a load balancer to decide if an instance it's managing is healthy or not
3. A way to configure the scaling system to gradually add new instances into the scaling pool, and rotate the old ones out
4. A desire to build something called Rolling Deployments of Auto Scaling Groups

My expectation would be that the scaling system (#2) would make full use of the health-check (#1) when deciding if a new instance has been launched successfully, before moving on (#3) to launching the next one. This would be a rolling deploy (#4), and one that works very hard to "increase availability during an update".

In concrete terms: when rolling the instances of an ASG that's connected to an ELB in an operation governed by an UpdatePolicy, I'd expect the ASG to use the full power of the ELB's health-check to decide when to move on to the next instance.

This doesn't happen. If expect it to, you'll likely experience some downtime when doing a CloudFormation UPDATE operation that replaces the instances.

It works how?

What happens instead is that the ASG adds the instance to the ELB immediately on it getting into the InService state (meaning the instance has booted). The ASG then ignores its own setting for HealthCheckGracePeriod, and starts counting the health-check failures (coming from the ELB health-check). This means that if the time your health-check settings will tolerate a non-responsive instance is less than the time it takes for your instance to be ready to serve requests (after it's done booting), the ASG will drop that newly-created instance. While this is happening, the UpdatePolicy rolling update will continue - which could very easily leave you with no working instances in the ELB.

As AWS support puts it:

Although you are using an ELB health-check, unfortunately as part of the rolling update process CloudFormation does not check that the instance has been marked as in-service behind the ELB. I do agree that when an environment is configured to use an ELB health-check, just relying on the fact that the instance has been added to the ASG is not a sufficient success criteria for rolling updates. There is currently a feature request logged with regard to this...there is currently no ETA.

And subsequently:

As soon as that new instance is started up and marked as in service in the AutoScaling group (fairly quickly), the ELB starts its health-checks.

What makes this even more frustrating is that all these pieces work together perfectly on non-CloudFormation auto-scaling operations. If one of my instances crashes the ASG will spin up a new one, add it to the ELB, and respect the HealthCheckGracePeriod during start-up.

Way to violate the principle of least-surprise there, AWS.

What now?

Getting past the disappointment this doesn't work, we still need a zero-downtime deploy solution.

The easiest thing to do is to extend the value for PauseTime in your UpdatePolicy to longer than it takes for an instance to move from a "booted" to "ready to serve web requests". Here, PauseTime governs the wait between adding in new instances and removing the old ones. That transition from "booted" to "serving" still needs to take less time than your "bad-instance" window (which comes from your health-check settings), but if it doesn't, you'll still have the old instances running to serve requests that come in.

The "official" solution is to use WaitOnResourceSignal, which pauses the rolling update until the instance uses cfn-signal to signal that it's in a good state - here's their example template.

In closing

This whole piece could be summed up with "it's bad that CloudFormation UpdatePolicy rolling updates don't respect the ELB health check", and it really is. I can't imagine the product decision that got AWS here, and I'm not staying up late to wait for the fix.

There's something funny in this, though. If you read the example template for how to work around this with cfn-signal, their example cloud-init configuration won't sent the OK signal until...a check on instance's ELB health (using the AWS CLI) comes back successful.

It's almost as if they knew how this whole thing should fit together, and then built something else.

At this point, you'll noticed that AWS has several products for this, which seem to all do almost the same thing. You've got:

• CloudFormation, which is the combination of a JSON DSL to create templates and a service to realize them
• OpsWorks, which is Chef with some AWS resources added into it
• Elastic Beanstalk, which is AWS' version of Heroku

But you've also got Terraform, which is unlike the others in that it's not built by AWS.

I've been using CloudFormation for about a year and a half, having built my own set of libraries and tooling using Troposphere - so I'm working in Python, and then converting to JSON at the end. I've been happy with this, but several of my colleagues have gotten excited about Terraform - which suggested it was time to give it another look.

Having spent a day rebuilding a simplified version of our current infrastructure on it, some thoughts:

The Good

• There was a lot less copy/paste than I expected. Python's list comprehensions are terse enough that I can condense many networking rules down into a few lines. While that's not an option here (no loops in a declarative language), the copy/paste required to get the same rules wasn't that annoying, and would be hidden behind the first layer of modules.
• Terraform's ability to undo things you've done in the console is outstanding. One of the weaknesses of CloudFormation is that it ignores any manual changes you've made to the infrastructure once it's been launched, whereas Terraform lets you reset any hand-configured settings back to what the code says.
• New AWS products (like NAT Gateways) get added into Terraform before CloudFormation, since Terraform uses their vanilla APIs. On the downside, there are a few AWS features that aren't exposed outside of CloudFormation - so Terraform can't use them.

The OK

• One of the big gripes we have with our in-house libraries is that building things takes a lot of proprietary knowledge. Once you've built up a tier of in-house Terraform modules, though, I suspect you'd end up in the same place.
• The documentation is about the same level as CloudFormation's, although I'd often find myself looking at the docs for the matching AWS function to really understand what a particular argument meant.
• Most of the cases where we change the system between production and development seem like they could be handled with variables. There are a few that don't have obvious solutions, but that could also be a sign that we shouldn't be doing that.
• The workflow (and some of the issues around apply) strongly suggests that Terraform is designed for blue-green deploys, which aren't always the right answer.
• Terraform uses the full "path" of a module to track it, meaning that refactoring some code that doesn't change the underlying infrastructure will cause a change so that Terraform can "keep up". Coming from Troposphere, I was used to the code being an abstraction that generated a plan at the end - as opposed to the code being the plan.

The Bad

• Several times I found that I had created a system that I couldn't destroy, which centered around this issue. Even following the work-around in there - which seems like a hack - destroy often found cycles which made it necessary for me to manually destroy bits of the system through the console.
• The design of the module system (and tooling - I'm looking at you, terraform get) gets frustrating when you're trying to use them for information-hiding and internal code-reuse, as opposed to for sharable units.
• Not being able to pass maps to modules got old. I had hoped to use a pattern of passing a mapping (e.g. "environment type to instance size") and a variable (e.g. "environment type") to modules and letting them figure out/hide which value to use; instead, this had to be done at the caller level.
• When there's an error executing the plan Terraform generates - which happened more than I expected - it doesn't roll back what it was trying to do. This is intentional (and clearly stated) and while I like it intellectually, it puts me at a higher risk of leaving my production system in a half-changed state, and my having to troubleshoot while I'm down. This is doubly worrying because the most common suggestion for how to fix this is "just run it again and see what happens". I much prefer CloudFormation's automated rollbacks, guaranteeing I end up in a known-good state on a failed change.
• Terraform doesn't support UpdatePolicy, so you need to use a bit of a hack to gracefully rotate new instances into an auto-scaling group. This isn't their fault - that API isn't exposed by AWS - but their solution isn't great, and doesn't really work on auto-scaling groups that aren't attached to load balancers.
• I'm still not sure how I'd handle stateful things like RDS instances in a Terraform world. There are all sorts of reports that this litters your .tfstate with credentials (although I didn't try it personally), and I'm not confident that the planner wouldn't one day try and destroy the instance as a side-effect to some innocuous change.

So?

After all that, I'm still glad I spent the day on Terraform. I used to think it was a bad idea; now I think it's a good idea, albeit one that's not yet ready.

The product is miles better than when I tried it a year ago, and it has - when they work - some seriously compelling features. But would I use it in production to manage a network? Not yet.

Things I did to make it less stressful:

• Used Augusta Movers. They weren't the cheapest of the companies I surveyed, but the owner was super-responsive, it was very easy to set up, the team was punctual and hard-working, and they did a great job packing things up. They also stressed in every email that you have to pay HST even if you're paying cash, which I appreciate. At no point in the process were there any surprises, which is about the highest compliment you can pay to movers.

• Ordered Frogboxes plastic boxes, as opposed to buying or scavenging cardboard ones. You're an adult now; you can do better going to the LCBO every day and getting Alberta Premium boxes. These things are clean, strong, convenient, water-repellent, and easy to move. You can put fragile things in them and not worry they'll be crushed, you can half-fill them and they'll still stack, and they make loading and unloading a breeze - they stack perfectly and uniformly. They'll end up costing a little more than cardboard boxes (~10% in my case), but not having to deal with the mess and cleanup of all that cardboard made it worth it in a second.

As a result of this, my father and I have spent several summers worth of testing all the vinegar-based^[1] coleslaw recipes we could find - and our favourite came from, of all places, about.com. It's not overly sweet, it goes (well) with everything you'd want to eat in the summer, and it can stand up to the strong barbecue flavours that come off the grill. It also ages well, which is handy for logistics.

Our coleslaw recipie, modified from about.com's Carolina Slaw:

Coleslaw

1 large head of cabbage, finely shredded
1 red pepper, finely chopped
1 medium onion, finely chopped
2 carrots, grated
1 Jalapeno pepper, finely sliced

1 cup cider vinegar
2/3 cup vegetable oil
1/2 cup white sugar
1 tsp. salt
3 tsp. celery seed
1 tsp. dry mustard powder
1/4 tsp. pepper

 

Combine all vegetables into a large bowl. Combine all dressing ingredients into a small pot, and simmer until everything except the celery seed has dissolved (about 10 minutes). Pour dressing onto vegetables and refrigerate for at least a day, stirring twice a day.

This is eminently edible after an hour or two in the fridge, but is at it's best after about two days.

If you're having surprise guests and need more, add some cabbage into what's left in the main bowl and stir it around some. You can also add in any similar fresh vegetables you have - I've had good results adding in half a head of thinly-sliced fennel. Lastly, if you'd like to take your shredding/chopping game up a notch, a mandoline makes quick work of all the vegetables here.

This is my go-to recipe when I don't want to try something new, only slightly adopted from Virtual Weber Bullet forums:

Miss Piggy Mustard Sauce

1 cup generic yellow mustard
1/4 cup white sugar
1/4 cup light brown sugar
1/2 cup cider vinegar
2 tbsp. chili powder
1 tsp. Cayenne pepper
1 tsp. ground black pepper
1 tsp. white pepper
1 tbsp. dark soy sauce
2 tbsp. butter

 

Combine all ingredients except soy sauce and butter, adding vinegar last (to desired thickness). Simmer for 10 minutes, then remove from heat and add butter and soy sauce.

A couple of notes, for when you're making it your own:

• I like to use Chipotle chili powder for a bit of a smokey taste, but try all the ones you can find to pick which one works for you.
• Traditionally, white pepper is used to not discolour sauces - which clearly doesn't apply here. You could almost certainly just use more black pepper and not miss much, but the original recipe called for both (so I use both).

Despite this being a mustard-based sauce (traditionally from South Carolina), it'll end up looking almost as dark as a ketchup-based on - but you'll definitely taste the lack of a tomato base.

Amazon recently published a set of best practices for building your AWS environment in a way to handle DDOS attacks. While the document is a great start and should be required reading when designing an environment, it's lacking in a few meaningful places - each section corresponds to a guideline of the same name.

Minimize the Attack Surface Area

Or put another way, "put the same diligence into designing your AWS network as you would a physical network." This is a great point and one which shouldn't be ignored, however the only pieces of advice in this section boil down to Use a VPC and Use one of the four VPC scenarios, which are as standard AWS advice as they'll give you anywhere.

So which scenario should you use?

AWS will rarely answer this question directly, because answering it in a way that makes sense for all systems would be really hard. But if you make a few simplifying assumptions - like I will momentarily - the answer becomes much easier. We'll assume:

1. You're building a pretty standard web application - some app servers, a few workers for async processing, a cache or two, and a database.

2. You don't need a hardware VPN, because your entire application will live inside AWS (or the parts outside can be reached via HTTPS/TLS).

That leaves scenarios one and two, which could be summarized as Give everything a public IP^[1] and Have a private subnet where your app servers & workers don't have public IPs^[2]. In an ideal world, the second solution is better. You can't attack a machine which doesn't have a world-routable IP, you can monitor everything going out of the private network on the NAT, security groups are striaghtforwards, etc. All of these points are true, but they miss one key thing (which I feel AWS does a poor job of highlighting):

AWS NATs aren't first-class objects.

The way they're mentioned in passing and have their own icon in AWS drawings^[3], you could be led to think that they're things you can create in CloudFormation, and maybe you just haven't yet found the docs for them. You'd be wrong if you thought that. AWS NATs are standard EC2 instances which you boot using an AMI provided by AWS, which are really just stock Amazon Linux machines that run a script on boot. AWS does no management of them for you above and beyond what they do on any other instance you launch.

This is troublesome, as it means that upgrading a NAT is now a downtime event for the app servers behind it (if they reach out to any services outside of your network). It also means that if you're going for an environment that manages itself as much as possible, you need to put that NAT instance into a auto-scaling group, and then do some scripting to update the routing tables when an instance boots^[4]. Lastly, if you're in a situation where for compliance (or other) reasons you need to keep an inventory of system components and installed software, these instances will fall under that regime.

So what do I recommend? Sadly, it's what AWS calls Scenario 1: put all your instances into a public subnet, and then use security groups to make sure that only specific boxes accept connections from outside the network. Your attack surface is larger (as everything has public IPs), but you've mitigated that risk with proper controls.

Be Ready to Scale and Absorb the Attack

Their advice here could be summarized as Use ELBs^[5] for your app servers, DNS which won't go down, and a CDN.

The DNS and CDN sections aren't worth much further discussion, aside from the fact that AWS CloudFront is only an OK CDN (I much prefer Fastly), and that there are other DNS providers out there beyond Route53 - although it does play extra-nice with ELBs.

Their advice about using an auto-scaling group behind an ELB is well-intentioned and correct, but mentions only in passing that [additional] instances may increase your costs - which is the biggest risk you face using this setup. Unless you're careful with your auto-scaling rules, an auto-scaling group will keep spinning up and spinning up instances in the face of a DDOS attack - and you'll lose if you try and outspend a DDOS attack.

Although they don't talk about it since it's not their product, this is the natural place where the document should suggest looking into vendors like CloudFlare or Incapsula to handle this for you.

Safeguard Exposed Resources

This section is mostly a grab-bag of other product offerings AWS would like you to use.

There's an extensive section about deploying web application firewalls on here, but that suffers from the same risks discussed earlier around trying to outspend a DDOS attach - if you put WAF nodes into an auto-scaling group (as it suggests), you'll spend all your money scaling up that group...as opposed to the app servers. Same result (the attacker makes you spend more, triggered by an auto-scaling rule), but a different and more expensive one, as WAF AMIs generally charge by the hour.

Again, the best advice to safeguard exposed resources would be to put them behind something like CloudFlare, so they're exposed to someone else's network and not yours.

Learn Normal Behavior

You should definitely do this, but I've never been satisfied by CloudWatch, their suggested way of doing it.

Does CloudWatch record all the things it says it records? Yes.
Will you have a good time dealing with that data? Nope.

If you're serious about understanding what your system is doing at any point in time, you should find a vendor/product that just focuses on that - they are both hard enough domains as-is. For performance monitoring, I love New Relic - it's not cheap, but monitoring instance health is the only thing they focus on, and I've never not felt like I've gotten positive ROI on it (with room to spare). For application-level logging I prefer LogEntries, but try the demos in this area - there are a few distinct approaches to this problem, and you should pick the one that matches your thinking.

Having said all that, I'll applaud AWS for saying you should put an alert on the EstimatedCharges metric, which tracks your estimated monthly bill. If you do nothing else, do this right now.

Create a Plan for Attacks

This is a great idea, but not in the way they're suggesting.

Sure, having an AWS TAM^[6] will help a little when you're in the midst of an attack (and paying for business-level support is a great investment, especially when getting started), the biggest return will come from having a plan in place before things happen, and having tested it a few times in a tabletop exercise. How this will work is very organization-specifc, but Heroku's incident response framework is a great start.

Conclusion

Building a system that's completely DDOS-resistant is impossible, and likely not worth the time even if it was.

As opposed to trying to beat script kiddies all day every day, you're much better off accepting that a successful DDOS attack^[7] will eventually happen, and coming up with a plan about what to do when that day comes. I'd start with making sure that all parts of your organization can agree on answers to the following questions, focusing on when you're being attacked:

• Who's in control internally, and what do we expect of them?
• How much of an attack do we expect to be able to stand up against?
• How open will we be with our users, and what will their experience be?
• How can other parts of the organization support the effort?

It's worth highlighting that none of the points above have much to do with technology, which is intentional. DDOS attacks are scary and stressful times, and your societal/interpersonal problems will be the dominant factors. Get those right - with practice, honestly, and communication - and you'll have the space needed to deal with the technical ones.

From Craig Butcher, whose site goes up and down:

1. Jump the directory where Ghost is installed -
   cd ~/webapps/appname/ghost
2. wget the newest release and unzip it - unzip -uo ghost-0.6.5.zip
3. Update npm - ../bin/npm install --production (and don't worry if you get a warning that forever should be installed globally with -g - WebFaction already did this for you)
4. Restart Ghost - ../bin/stop && ../bin/start

This is the story (slides) of how pressing "Enter" on a curl command gets you something back.