<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <title>Josh Brody</title>
  <subtitle>Federal prison alumnus, product engineer,  and writer when I have something to say. These are all related.</subtitle>
  <link href="/atom.xml" rel="self" type="application/atom+xml"/>
  <link href="/" rel="alternate" type="text/html"/>
  <updated>2026-07-20T14:29:52+00:00</updated>
  <id>/</id>
  <author>
    <name>Josh Brody</name>
  </author>
  
  <entry>
    <title>Reflections on presenting at InterCOP, hosted by INTERPOL</title>
    <link href="/posts/reflections-on-presenting-at-intercop-hosted-by-interpol/" rel="alternate" type="text/html"/>
    <id>/posts/reflections-on-presenting-at-intercop-hosted-by-interpol/</id>
    <published>2026-07-18T00:00:00+00:00</published>
    <updated>2026-07-18T00:00:00+00:00</updated>
    <content type="html"><![CDATA[<h1 id="reflections-on-presenting-at-intercop-hosted-by-interpol">Reflections on Presenting at InterCOP hosted by INTERPOL</h1>

<p>In July 2026 I presented at the 7th International Cyber Offender Prevention conference: InterCOP, hosted by INTERPOL in Lyon, France. Speakers talked about a very specific, single premise: the cyber threat is getting younger and faster, and we need better ways to stop kids before they become cases. Prevention. Interception. “Let’s get ahead of the thing instead of cleaning up after it.”</p>

<p>It’s a good premise. All speakers delivered on it. There was one notable missing delegation: mine.</p>

<p>Edit: Even more damming, between my well-connected co-presenter and I, neither of us found another American at the conference.</p>

<style>
.gallery {
  display: grid;
  grid-template-columns: repeat(auto-fill, minmax(150px, 1fr));
  gap: 12px;
}

.gallery img {
  width: 100%;
  aspect-ratio: 1 / 1;
  object-fit: cover;
  border-radius: 4px;
}
</style>

<div class="gallery">
<img src="/assets/images/posts/reflections-on-presenting-at-intercop-hosted-by-interpol/1.jpeg" />
<img src="/assets/images/posts/reflections-on-presenting-at-intercop-hosted-by-interpol/2.jpeg" />
</div>

<h2 id="the-room-was-good">The room was good</h2>

<p>The real prevention work came from people who came with a method and the scars from running it. Huge credit to the Dutch who are leading these efforts (and it’s not even close, but that’s a testament to what they’re doing and trying). Their work is genuinely interesting, and it’s clear that their government supports their efforts.</p>

<p>Some notes on the content, generally:</p>

<ul>
  <li>Behavioral intervention models that are structured and built to plan exactly where and how you interrupt a criminal phenomenon</li>
  <li>Comparative work on why one kid who loves computers ends up offending and another with identical interests doesn’t</li>
  <li>National survey data mapping the early, still-legal stretch of the pathway where you can actually reach someone</li>
  <li>Structured deterrence programs with years of operational history behind them</li>
  <li>Gamified interventions aimed at literal elementary schoolers</li>
</ul>

<p>This was presented by European teams, mostly. Some Nordic, some further afield. The field is better for their efforts.</p>

<p>Almost every one of them had a government behind them. National police, or bureaus of investigation. Some agency with a prevention mandate and the budget to chase it—things unheard of in America. I didn’t see one consultant selling anything. It was people talking about government work—because they <em>were</em> the government, standing up to report what their country is actually doing to get ahead of youth cybercrime.</p>

<p>That’s the part that stuck with me. Not the quality, but the staffing.</p>

<h2 id="loud-sigh-big-yawn">Loud sigh, big yawn</h2>

<p>I sat through talk after talk and eventually found a pattern: accents unfamiliar to me. I opened the agenda and looked for Americans. Homeland Security Investigations was teaming up with a Dutch researcher; a vendor was reporting intel on online communities.</p>

<p>Then there was me and my FBI profiler: a guy who ran illegal infrastructure and went to federal prison for it; a person who did the work inside the government on my case. Two ex-feds on different wavelengths: offender and employee, currently operating outside the machine. Inside the machine, this work apparently doesn’t have a seat.</p>

<p>I don’t say that to knock the Americans who were there doing other things—the investigative and intelligence work was genuinely strong. But it was investigation, attribution, understanding the ecosystem. Every other country in that room sent people to talk about stopping the thing before it starts. My government’s contribution to that specific conversation came from two men with no backing between them, let alone an endorsement.</p>

<h2 id="why">Why</h2>

<p>Here’s my theory, and it’s no more than a theory—I have a hunch and an argument.</p>

<p>US cyber enforcement is built around prosecution as the goal. Everything upstream feeds charges. Attribution, intel, the whole apparatus—it points at a courtroom. That’s it. A big, well-funded, genuinely capable—albeit imperfect—system.</p>

<p>There is nothing in that pipeline for prevention. The best possible outcome in prevention is a kid who never becomes a case. No arrest, no press release, no sentence. Points are scored by getting convictions; an intervention that produces nothing to charge looks, from the inside, like doing nothing.</p>

<p>The programs I watched come out of places where “we redirected them” is a win—fundable, reportable, good for a career. In the US, the win condition is a conviction, and a conviction requires an offense. You have to let the thing happen to score. No defense other than the defendant.</p>

<p>That’s the structural defect, and here’s the icky cultural version sitting on top of it: Americans are perpetually horny for justice—just look at our top TV programming. We would rather catch and punish the bad guy than quietly arrange for there to be no bad guy to catch. The former feels like righteousness and the latter feels like a rounding error.</p>

<p>One of them makes the news. (Disclaimer, potential bias: I very much made the news.)</p>

<p>I could be wrong. Maybe the US government prevention people exist and simply weren’t present—the field is young, and who gets on a plane to Lyon is half accident. I’d believe that. But something has to explain why the country with the largest cyber enforcement apparatus on earth showed up to a prevention conference with no prevention arm to speak of, and left the seat to be filled by an ex-con and a guy who quit.</p>

<p>Big laughs to one of my well-resonating bits:</p>

<blockquote>
  <p>Shoutout to every country that made it past the round of 16 at the World Cup. You all have something in common: government-run healthcare, and governments that give a shit about cybercrime prevention and intervention.</p>
</blockquote>

<h2 id="what-we-brought-instead">What we brought instead</h2>

<p>For the record, the seat did get filled.</p>

<p>I presented with the FBI profiler who was on my case. We’re friends now and I refer to him as <em>my FBI profiler friend</em>. Our work presented a simple, uncomfortable idea: don’t destroy offender infrastructure, but disrupt it. Engineer small, deniable failures instead of publicly seizing the thing. A takedown ends the story: <em>this domain has been seized</em> is hardly effective but sexy for a press release. A minor, well-placed malfunction keeps it running and tells you enormously more—how the offender responds, what they trust, what they value, and where they’re vulnerable. Minimal interference is more behaviorally exploitable than a raid.</p>

<p>That’s an applicable method. It’s a deliverable the room asked for.</p>

<p>And it came from having been the offender on the receiving end of exactly this—not from modeling it from a desk or doing surveys. I ran the infrastructure, understood the scene, the demand, the want and the need. I also did this at a large scale. I felt what a disruption does to the person operating under it, because I was that person. You can profile that response from outside. It’s not the same as having lived inside it.</p>

<p>I’ve come to understand that I’m rare candy in these rooms. There’s legal exposure and reputational risk. I’d gather it’s weird to mail an invitation to the former prisoner who is fresh out of prison and just starting probation. But I’d argue that’s why it’s worth something.</p>

<h2 id="what-prevention-loses">What prevention loses</h2>

<p>The US is not short on threat <em>description</em>. It has plenty. It is not short on investigative muscle—it has more than it knows what to do with and wastes its time regularly.</p>

<p>It’s short on prevention, at the government level, in a room where many countries were represented. And it’s shortest on the one perspective that turned out to have real value: someone who walked the pathway and came back to explain it.</p>

<p>Every other delegation had a government behind them. Mine had two guys who did this in spite of one. That should bother somebody in Washington, but it probably won’t.</p>

]]></content>
    
    <summary>Reflections on Presenting at InterCOP hosted by INTERPOL
</summary>
    
    
    <category term="cyber"/>
    
  </entry>
  
  <entry>
    <title>Behavior</title>
    <link href="/posts/hard-determinism-as-religion/" rel="alternate" type="text/html"/>
    <id>/posts/hard-determinism-as-religion/</id>
    <published>2026-06-17T00:00:00+00:00</published>
    <updated>2026-06-17T00:00:00+00:00</updated>
    <content type="html"><![CDATA[<p>Robert Sapolsky’s <a href="https://en.wikipedia.org/wiki/Determined:_A_Science_of_Life_Without_Free_Will">Determined</a> changed how I think about human behavior.</p>

<p>Human behavior has largely been shaped by religious beliefs, its preachings and lore, its books and the things only explainable by sciences we hadn’t yet captured reasonably.</p>

<p>If every thought, choice, and action is the inevitable product of prior causes—genes, hormones, childhood, culture, the neuron that fired before you had any say over it, your conception which you didn’t author and its snowballing effect, the millenniums of history that shaped all of it—then there is no uncaused causer. Nothing in the brain steps outside the chain. This is the soul, and the first tenet is that it doesn’t exist.</p>

<p>The serial killer and the saint are both suddenly equal based on the output of a luck they never wished upon, then no one authored themselves. Praise the weather; how dare the tide come in.</p>

<p>Suddenly contempt and hatred and retribution fall apart in your hands. The person you can’t stand was assembled by forces they didn’t pick, just as you are. You stop asking what’s wrong with them and start asking what happened to them.</p>

<p>If you can’t hate a person for what they are, what’s left to do with the dangerous ones? Failing brakes mean you take it off the road, but you don’t despise it. Lock the danger away if you must, but the wish for them to suffer for being who the universe made them doesn’t earn its keep.</p>

<p>So what happens to wonder? You can still feel awe at the universe, gratitude when things break your way, love for the people around you. None of it needs anyone to have freely chosen anything. Feelings are caused too, and being caused doesn’t make the result any less real.</p>

<p>You can’t shake the feeling that you’re choosing. It’s baked in too deep for that. So you believe one thing and feel another, like how the Earth is tearing through space and still sits dead still under your feet.</p>

<p>You won’t choose to believe any of this either. You’ll be persuaded or you won’t, and that was never up to you. Isn’t it ironic, don’t you think?</p>
]]></content>
    
    <summary>Robert Sapolsky’s Determined changed how I think about human behavior.
</summary>
    
    
    <category term="personal"/>
    
  </entry>
  
  <entry>
    <title>No more asking &apos;who are you&apos;; we have all been reduced to a passphras</title>
    <link href="/posts/we-have-all-been-reduced-to-a-passphrase/" rel="alternate" type="text/html"/>
    <id>/posts/we-have-all-been-reduced-to-a-passphrase/</id>
    <published>2026-05-30T00:00:00+00:00</published>
    <updated>2026-05-30T00:00:00+00:00</updated>
    <content type="html"><![CDATA[<h1 id="what-passphrase-are-you">What Passphrase Are You?</h1>

<p>I went to prison. I don’t hide this. The shame has passed.</p>

<p>While I was gone, someone stole everything I owned in the physical world and then, for good measure, changed the passwords on most of my digital life. This included my GitHub account, moved my domain name across three registrars, my Google Workspace account tied to that domain name (as had been for 13 years), 2FA keys—you know, just about everything. I’ve gotten most of my digital accounts back—I’ve lawyered up for the personal stuff (some of which has been posted on Facebook Marketplace). I had backup codes printed on paper, but predictably that paper is gone, the passcodes are forgotten.</p>

<p>And so, I am forgotten.</p>

<p>I’ve spent the last several months proving I’m me. Over and over. To strangers.</p>

<p>In the name of security, we have removed ourselves as humans. We have redefined what a person is. A person is no longer a human being with a face, a name, a history, and a stack of corroborating evidence. We are not even DNA (can’t I just send you a sample?).</p>

<p>No, a person is a passphrase.</p>

<p>If you have the string, you’re you. If you don’t, you’re nothing—no matter what else you can produce.</p>

<h2 id="what-i-could-prove">What I could prove</h2>

<p>The abstract version of this sounds like whining and the specific version is the actual problem.</p>

<p>For GitHub: I had the original phone number on the account. I could name endless private repos. I could tell you the commit count on one of them—6,992. I could tell you hardcoded strings in that repo. I could identify myself as the defendant in the case which GitHub received subpoenas for from the federal-fucking-government.</p>

<p>None of it was enough. The ticket response was a polite version of “we can’t confirm you are you.” I opened it; it went nowhere.</p>

<p>For my domain, josh.mn—held since 2012—I got it back. Obviously.</p>

<p>For Facebook, Instagram, my Google Workspace, the rest of it—I got those back too. Eventually. By grinding.</p>

<p>Had those accounts <em>not</em> been mine, the same recovery process I muscled through would have handed an attacker the keys to drain bank accounts, harvest identities from email recovery flows, and push a supply-chain attack through two packages that other people’s code depends on. The recovery process is the attack surface. It worked for me. It would have worked for someone pretending to be me.</p>

<p>That’s not a hypothetical. That’s the literal mechanism. The same door that let me back in is the door I’d have to defend against if I were anyone else.</p>

<h2 id="to-the-passcode-it-may-concern">To the passcode it may concern</h2>

<p>Apple. Fucking Apple.</p>

<p>My late mother’s photos and documents are in an account I can identify down to the serial numbers on the devices—including the serials <em>the federal government documented for me</em>, because of how I ended up where I ended up. I have the email address associated with the account (okay, easy). I have more than 10 serial numbers associated with devices that have been on the account (less easy). I have a prison ID (ain’t ever seen someone trying to fake this one). I have my ID, my passport. I have a demand letter sent to the person who stole my property, their IP address, the date they changed the password, the date they deleted original passphrases. I have the date they removed my iPhone from my account—you know, all these dates were when I was in prison. (Caveat: not like it couldn’t have been done from prison, but it just makes it a little more harder to do.)</p>

<p>I can produce more corroborating evidence than most people could assemble for their own existence.</p>

<p>This shitshow cascades: I entrusted <em>all</em> my belongings—including my physical devices—to someone. She transferred my phone number out to a different carrier, changed my account’s passphrase, stripped my numbers off the account, and added hers. By the time I was out, the account had been quietly rebuilt around her and locked behind a string I never set, to a phone number I never controlled. I didn’t lose the passphrase. It was taken, along with everything else.</p>

<p>So that’s it. The answer is no. Not “let’s verify through another channel.” Not “let’s escalate to a human who can weigh the evidence and make a judgement call.”</p>

<p>Just no.</p>

<p>The passphrase is the person. I am not the passphrase. Therefore I am not the person.</p>

<p>And so the photos of my mother sit behind a string I can’t reproduce, and every other fact about me is deemed irrelevant.</p>

<h2 id="jesus-christ-would-be-just-as-screwed-apparently">Jesus Christ would be just as screwed, apparently</h2>

<p>I lobbed this to multiple people who were assigned to my Apple “account recovery” case. Circa year 0, with iPhones and shit:</p>

<p>Jesus Christ gets mugged on his way to a river walk: his phone and wallet taken. While he’s on the chariot to the hospital, he’s SIM-swapped. He had a 2FA key in his wallet; the backups were written down in ink-ish stuff on a rock at his residence—no, not carved, sorry.</p>

<p>In Apple’s eyes, Jesus is no longer Jesus. Biometric records, a paper trail thousands of people could attest to—none of it counts. “Yeah, you’re Jesus, but you don’t have your passphrase, so you can’t be you.”</p>

<p>Not a person.</p>

<p>We’ve built a system where identity is a single point of failure and there is no human in the loop to catch the catastrophic case. “We have policies in place.” Those policies are made by reasonable humans who expect reasonable things to happen to reasonable people.</p>

<h2 id="but-the-alternative-is-account-takeover">“But the alternative is account takeover”</h2>

<p>Yeah, no shit. It’s a fair reflexive objection.</p>

<p>Yes—a recovery path that weighs <em>evidence</em> is a recovery path an attacker can try to walk. Social engineering is exactly this! Every “verify your identity” flow that isn’t a cryptographic secret is, in principle, defeatable by someone who collects enough of your facts.</p>

<p>But notice what the companies actually do. Google, Facebook, my domain registrar—the evidence path <em>exists</em>. It’s painful, it’s inconsistent; sometimes it’s staffed by people who’d rather say no than be the one who let an attacker in. But it exists, and it can succeed, which is precisely why it’s also an attack surface.</p>

<p>For GitHub, an affidavit that includes language of <em>if this isn’t you we’re going to tell your probation officer</em>. (I am not joking.)</p>

<p>Apple’s position is the opposite. There is no path. The passphrase or nothing. They’ve eliminated the social-engineering risk by eliminating the human entirely—and in doing so they’ve also eliminated me.</p>

<p>So the industry doesn’t actually agree that evidence can’t be weighed. Half of it weighs evidence badly and inconsistently. The other half refuses to weigh it at all. Neither of those is a designed answer. They’re just two different ways of not having solved the problem.</p>

<h2 id="the-part-nobody-wants-to-own">The part nobody wants to own</h2>

<p>The honest version is this: weighing identity evidence at scale, for free, against motivated attackers, is genuinely hard. It costs money. It requires trained humans making judgment calls and eating the occasional expensive mistake. So the cheap move is to declare the passphrase sacred and the human irrelevant, and call it a security posture.</p>

<p>Liability-as-a-disservice under the guise of security.</p>

<p>Real security would say: a single secret should never be the only thing standing between a verifiable human and their own data. There is human error there.</p>

<p>Real security would build a graduated, evidence-weighing, human-reviewed path for the catastrophic case—device serials, government documentation, corroborating accounts, biometrics, an in-person option—and would price that path honestly instead of pretending it can’t exist.</p>

<p>In a thread announcing FTPgod drewh is stepping down CEO of Dropbox on Hacker News—on a thread about him <em>retiring</em>—<a href="https://news.ycombinator.com/item?id=48284358">I posted a comment begging for help</a>, twelve days after I’d already emailed him (I have read receipts, drewh!). He reached out personally. That worked. I’m grateful.</p>

<p>But “get a billionaire’s attention on a news thread” is not an identity recovery system, just as HackerNews should not be every company’s support channel but it sure seems that way sometimes.</p>

<h2 id="so-who-are-you">So, who are you?</h2>

<p>It’s no longer “who are you?” instead it’s “what’s the string?”</p>

<p>A human is not a passphrase.</p>

<p>The moment we forgot that, we built systems that work perfectly right up until there’s an edgecase of an edge case. Nobody ever thought <em>what if Jesus gets mugged, SIM-swapped, and his house torched?</em></p>

<p>I got most of my life back by being stubborn and, in one case, lucky. My Apple account is still locked behind a passphrase never set by me.</p>

<p>I’m just a passphrase.</p>
]]></content>
    
    <summary>What Passphrase Are You?
</summary>
    
    
    <category term="personal"/>
    
  </entry>
  
  <entry>
    <title>The moat was never the code</title>
    <link href="/posts/the-moat-was-never-the-code/" rel="alternate" type="text/html"/>
    <id>/posts/the-moat-was-never-the-code/</id>
    <published>2026-05-21T00:00:00+00:00</published>
    <updated>2026-05-21T00:00:00+00:00</updated>
    <content type="html"><![CDATA[<h1 id="the-moat-was-never-the-code">The moat was never the code</h1>

<p>I ran an unauthorized sports streaming service from 2016 to 2021. It was called HeheStreams. I charged $125 a year; users could have a single annual payment of $100. My infrastructure cost was $75 a month.</p>

<p>HeheStreams didn’t re-encode or re-broadcast streams. My users consumed them directly from licensed platforms—using the same CDN and DRM as the platform’s paying users. I implemented more than a dozen different platforms in the name of feature parity across each sport.</p>

<p>People who know the technical side assume that was the hard part. It was hard. It was also the part anyone else could reasonably try to copy. I know, because I watched them try.</p>

<p>The thing that actually kept the lights on was customer service and device compatibility. “I could do that in a weekend” is a thing. I made my proof of concept in an hour in 2016. Was that scalable to a form a semblance of a business? No, but…</p>

<p>Here’s what parts don’t make it into that line, at least for me—the pricing and the support. And, while we’re at it, why I didn’t care if people copy me. Please do, it means I’m doing something right.</p>

<h2 id="a-widely-used-protocol-and-seven-lines-of-javascript">A widely used protocol, and seven lines of JavaScript</h2>

<p>I’ll keep the technical part short, because it’s the least interesting thing here, and that’s the point.</p>

<p>The dominant streaming shit when HeheStreams started was <a href="https://en.wikipedia.org/wiki/HTTP_Live_Streaming">HLS</a> with a <a href="https://mangui.github.io/flashls/">Flash-based player</a> as the client. With HLS, video gets chopped into short encrypted chunks (think mp4). To protect this, you can encrypt it with a key. WHen you do, the player fetches a key to decrypt them. The key lives at a URL written into a playlist file, and the player asks for it with an ordinary HTTP request.</p>

<p>And that’s what’s in the protocol, people.</p>

<p>The system assumes the player is honest about which key it requests. If you—dangerous cybercriminal—intercept the request in the browser, point it at an endpoint you control, you can ultimately return what you want instead of what someone else’s server wants. In practice that’s about seven lines of JavaScript wrapped around the browser’s own request function, sent to a request handler that can be a single line of code.</p>

<p>This was the proof of concept I built in 2016. When I saw it work, I shit my pants. Being an avid consumer of pirated streams, I can say with confidence I was the first person in the scene to do it. My code—verbatim, line for line, with a very specific caffeine-and-anxiety-induced function name—is still visible today on a few popular streaming sites.</p>

<p>I never obfuscated any of it. My client-side code was right there for anyone to read. It’s reasonable to ask why. “Protect your castle” doesn’t apply when you’re not driven by anything but sharing. But in a world of free, almost-just-as-good ad-supported options, seven lines of JavaScript wasn’t my moat.</p>

<h2 id="what-i-was-actually-selling">What I was actually selling</h2>

<p>Around that same time, most providers moved to DRM-backed streaming, which meant the days of seven lines of JavaScript became moot. My technical skills, and my skills as an offensive researcher is what allowed me to push the envelope and end up with a bunch of fed charges.</p>

<p>At the end of the day I was selling brand, something working on the device in your living room during a game your team was losing, and a human answering when it didn’t.</p>

<p>This sounds easy until I tell you the bad news for a developer: DRM is device-specific and it’s a mess; this cascades down to streaming format. Widevine, PlayReady, FairPlay—three systems, and which one applies depends on the device, sometimes the device and the browser. Widevine worked in Chrome on macOS but not Safari. Anything on iOS needed FairPlay. It gets worse from there.</p>

<p>Multiply that across every device, browser, and app combination a customer might own, plus the number of streaming platforms I needed to support, and there were over a thousand distinct configurations—with different code, different vulnerabilities, and different setups, because each platform decided to do things differently. Awfully so, sometimes.</p>

<p>That’s the real engineering. Not the key swap—the reality of “it works on my phone but not my TV.”</p>

<p>The technical implementation of my proof of concept was the lowest-hanging fruit. The actual work was everything around it.</p>

<h2 id="customer-service-when-the-customer-service-is-just-you-and-also-you">Customer service when the customer service is just you, and also you</h2>

<p>It was me.</p>

<p>Every ticket, every reply. I answered them personally and I answered them like a person—conversational tone, no <code>no-reply@</code>, everything signed off with some version of “if you need anything just reply :)”</p>

<p>I was proactive instead of reactive: I watched new accounts’ streaming activity and reached out if it looked like someone wasn’t set up right, before they thought to complain. Similar triggers existed for those who experienced client-side errors—I’d kick back a notification to my server and queue up an email. This proactive handholding is literally where one of my <a href="https://github.com/joshmn/caffeinate">open-source projects</a> came from. I built tooling to babysit new users and it outlived the site.</p>

<p>The whole customer-service-with-personality thing is hard to do at scale. I’ve seen it done if you allow people to be themselves, hire them for knowledge, and equip them with domain knowledge. Works best for B2C. B2B might smell different.</p>

<p>An aside, kind of related: I’m at a Mormon Easter dinner with a customer—long, hilarious story with the expected above-PG13-elements. We’re sitting on their parent’s perfectly manicured lawn, everyone in their Sunday best but me who didn’t ever expect to go to a Mormon Easter—I’m in my usual skin-tight black skinny jeans and a black t-shirt (which didn’t exactly fit for the day of resurrection). Eventually they got to talking basketball, and streaming. My ears perked up. Then they said HeheStreams, and I became the blinking meme.</p>

<p>I excused myself to go to the bathroom so I could politely search my users by email and check on some last names. I found 4 of the 7 adults at the dinner party with active subscriptions. Cool. Over dinner, they’re all cheerfully talking shit about my Roku app. Fuck.</p>

<p>They had no idea I was me. She did obviously—my customer-date-thing. They didn’t.</p>

<p>Roku was the one platform I could never fully commit to—the only one I offered “unofficial” because I hated the language I was forced to use (BrightScript). I made this very clear on every page that had the word <code>Roku</code> on it: something akin to “this is unofficially supported, may break, don’t rely on it, it sucks” etc. I don’t have stats now, but my guess is that it was also the most popular app that my users used (my service was available on everything from your browser to your smart fridge to your gaming console).</p>

<p>And so there’s a family who pays me every year, who’d presumably tell you they like the service, sitting around a dinner table eating roasted animal while roasting the shit I didn’t really care about but did care about at the same time. Which is cool, they’re customers, but what felt like resentment just roasted my ego.</p>

<p>There was a “magical” drawing for gift credit that day. My customer-date-thing’s brother won. She said it was nice. I said it was because they didn’t like the Roku app. You could split the gift credit as many ways as you wanted. He gave it to his folks and siblings.</p>

<p>Customer service is whether the thing works and whether someone picks up the phone.</p>

<p>To put a fun, exclamation point to the story:</p>

<blockquote>
  <p>Mom: So, Josh, how did you meet $customer?</p>

  <p>Me: Jazz Twitter.</p>

  <p>Mom: Oh! So you’re from here then? :)</p>

  <p>Me: No.</p>

  <p>Mom: Oh, donde are you from?</p>

  <p>Me: Minnesota.</p>

  <p>Mom: Then why are you on Jazz Twitter?</p>

  <p>Me: I was a big fan of the Deron Williams/Andrei Karalinko/Carlos Boozer/Kyle Korver teams (this wasn’t a lie; but I was on Jazz Twitter because they were my biggest customer base)</p>

  <p>Mom: Oh.</p>

  <p>Mom: So you must have family here?</p>

  <p>Me: No.</p>

  <p>Mom [to husband]: Honey, is there a big Mormon population in Minnesota?</p>

  <p>Dad: I don’t think so.</p>

  <p>Mom: Are you mormon, Josh?</p>

  <p>Me: No.</p>

  <p>Mom: <em>smiles in disappointment at her daughter</em></p>
</blockquote>

<p>It doesn’t end here, but for the sake of this blog, we will.</p>

<h2 id="pricing-out-customers">Pricing out customers</h2>

<p>I charged $125 a year, or $100 if a user decided they trusted me enough with their money to do a year up-front. That sounds like a lot until I tell you what the competition charged.</p>

<p>IPTV services—thousands of channels, movies, TV shows, with liberal account sharing policies— went for about $4 a month. I was several times more expensive for watching a single sport and a strict no-sharing policy:</p>

<p>This was intentional. Granted, the goal of my site was never to make money. Had it been, I would have actually marketed it—I didn’t, not because I was “scared” to draw attention to it (I laugh when people say these services “hide”).</p>

<p>I could have charged more. My users weren’t price sensitive, their choice to use HeheStreams was rarely about price (I surveyed this data, and even tested it at one point), and I’m confident I’d have kept at least 65% of them through an increase (my survey said almost 90%). I certainly could have charged less—my hosting bill was $75 a month, HelpScout was $22/month, Mailgun was less than $20/month. I could—and did—bill twice my monthly infra cost in a single hour of contract work. There’s more to pricing psychology than just this, but my goal was to have more savvy customers, and savvy tends to trend upward in society.</p>

<p>I priced myself out of more customers, which would have meant more money. The customers I wanted weren’t at the bottom. I tried to be like Apple, which sounds gross when you say it out loud and it’s true anyway: people respect what they pay more for. Someone who sees their subscription as an investment is patient when something breaks. Someone who paid $4 walks the second it hiccups, talks shit online with a shrug: “oh well, it was only $8.” The cheap customer is the expensive customer.</p>

<p>The gauge of affluence came from surveys, support tone—affluent, patient, treating me like a person they’d chosen rather than a deal they’d lucked into (which is why I never did discounts other than my annual). I also derived demographics from IP-to-zipcode, and making a twice-yearly investment into some creepy-as-fuck-identity-resolution stuff. I wasn’t guessing.</p>

<h2 id="so-yeah-steal-my-shit">So yeah, steal my shit</h2>

<p>Now back to the obfuscation question, because the pricing addresses it.</p>

<p>If you think the product is the seven lines of JavaScript, hiding the code makes sense. But the trick was never the product. Everyone else that cared had the trick—they’d copied it off me and I can still see it today. I could tell who was trying to get my other tricks because I logged right-clicks and devtools opens, and a site operator casing my site is painfully obvious: the same account opening devtools repeatedly, session after session, day after day for weeks.</p>

<p>I knew who they were. Their server IPs would eventually hit my site—which I’d quickly break my site for after a few automated requests (to protect my home), leaving their account’s IP in-tact (to confuse the hell out of them, and to be kind considering they gave me money). I could often connect a specific account to a specific competing site by IP and what I knew of that site—the operator of a Moroccan free-streaming site, for instance, was browsing my platform from a Moroccan IP address.</p>

<p>For these platform operators who were having problems, I’d email them. Not as a user—but as HeheStreams. “Hey, HeheStreams here. I saw on reddit that people were having issues with your site. I cruised around [platform they were trying to use]. Have you tried [this]?” I never said I knew exactly who they were on my own site. I didn’t need to. The point was made.</p>

<p>Obfuscating the code would have protected the one thing that wasn’t correlated to my ego. My ego was in pride in what I made and the customers who swore by my product.</p>

<h2 id="for-legitimate-things">For legitimate things</h2>

<p>Your copyable thing and your defensible thing are usually not the same thing, and you will spend most of your anxiety on the wrong one. This has risen exponentially: for all the vibe-coded slop, there’s the reality that someone else can, you know, also vibe-code it.</p>

<p>The code, the clever trick, the architecture—that’s the part you can see, so that’s the part you think to protect. It’s also the part a competent competitor reproduces in a weekend. What they can’t reproduce is the boring compound stuff: the long tail of edge cases you’ve actually fixed, the support, your brand reputation, the pricing that selected for patient customers instead of cheap ones.</p>

<p>My “startup” wasn’t something I’d ever recommend anyone run. The lesson is the same though: an investment in guarding the trick is a trick itself.</p>
]]></content>
    
    <summary>The moat was never the code
</summary>
    
    
    <category term="startups"/>
    
  </entry>
  
  <entry>
    <title>statement_timeout 0: Dirty plates, dead cooks, and kitchen fires</title>
    <link href="/posts/statement-timeout-0-casually/" rel="alternate" type="text/html"/>
    <id>/posts/statement-timeout-0-casually/</id>
    <published>2026-05-14T00:00:00+00:00</published>
    <updated>2026-05-14T00:00:00+00:00</updated>
    <content type="html"><![CDATA[<p>so you got tired of alerts that your queries were timing out. that’s bad because alerts are good even though nobody likes the alerts. my mom didn’t like acknowledging she might have cancer either so she just ignored the headaches and vertigo and balance issues and then it was like <em>ope stage IV brain cancer.</em></p>

<p>the name makes it sound like a fix. it’s not. removing the timeout doesn’t make the slow queries faster, it just lets them run forever, and “forever” on a 1tb db with bad queries is how you lose the restaurant.</p>

<p>let me explain what i mean by restaurant.</p>

<h2 id="the-kitchen">the kitchen</h2>

<p>think of postgres like a restaurant’s kitchen. food goes out as it’s done, not waiting for the plates to be done. the house rule is that if a cook takes more than 5 minutes (<code>statement_timeout</code>) on an order, kill them (<code>pg_terminate_backend</code>) and tell the customer they died but we got a new one already. the kitchen stays functional.</p>

<p>above is what happens when u dont kill the cook.</p>

<p>and the side effects compound. it’s not “one bad query takes a long time, oh well.” it’s “one bad query slowly kills every other query in the kitchen.” here’s the order things go bad in.</p>

<h2 id="the-slow-order-doesnt-get-faster">the slow order doesn’t get faster</h2>

<p>this is obvious. it’ll just sit on the stove.</p>

<p>but whoever is cooking it can’t take other orders cuz they only get one burner. each burner is a backend process—pg forks a process per connection, and that process is pinned to whatever query it’s running until that query finishes or gets killed. because kitchen has a fixed number of burners (<code>max_connections</code> / pgbouncer pool size), once they’re all stuck on slow orders, every new customer waits at the door.</p>

<p>from outside it looks like the restaurant is closed even though it’s “running normally”—<code>pg_stat_activity</code> shows like 6 active queries and you go “wtf the db is fine” but the app is throwing connection pool timeouts because there’s nothing left to hand out.</p>

<p>this is ok because we have 1000 burners and 1000 cooks. but that’s not really okay—but we’ll pretend like it’s okay because i’m more concerned about other things.</p>

<h2 id="the-busser-is-locked-out">the busser is locked out</h2>

<p>because nothing has left the kitchen, the dishwasher is just chillin’ on his phone waiting to do something.</p>

<p>pg has a background process that cleans up after itself—a busser clearing plates (autovacuum). when you <code>UPDATE</code> or <code>DELETE</code> a row, pg doesn’t actually remove the old version, it just marks it dead and leaves it there. mvcc. autovacuum comes through later and reclaims the space.</p>

<p>but the busser is only allowed to clear plates from tables where <em>everyone</em> has already left—specifically, dead tuples newer than the oldest running transaction’s <code>xmin</code> can’t be cleaned up, because that transaction might still need to see them. if one table has been sitting there for 3 hours (one query running for 3 hours), the busser can’t clear <em>any</em> plates from the last 3 hours, anywhere in the restaurant.</p>

<p>dirty plates pile up. the cooks have to walk around them. everything gets slower. the slower it gets, the more tables stay seated for hours, the more plates pile up. we’ve been in this death spiral before; it took weeks to recover from.</p>

<h2 id="why-your-1-second-query-suddenly-takes-a-minute">why your 1-second query suddenly takes a minute</h2>

<p>when the busser can’t keep up, every table has dirty plates stacked on it. a waiter who used to grab one clean plate now has to dig through 60 dirty ones to find it.</p>

<p>nothing about the waiter changed. the plate they want is still there. but the pile on top of it grew.</p>

<p>or, translated: a <code>SELECT COUNT(*)</code> that used to scan 10k live rows now scans 10k live rows plus 600k dead ones pg can’t throw away yet. it has to read every dead tuple, check the xmin/xmax to figure out it’s invisible to this transaction, and skip it. (the “heap” is just the pile of rows on disk for a table—live and dead mixed together. pg has to walk the whole pile.) same query, same code, same indexes—60x slower because the heap is full of bloat nobody’s allowed to vacuum while the long query is running.</p>

<p>and it’s not just the <em>bad query</em> (the one you “optimized” using statement_timeout) that gets slow. it’s everyone’s queries against that table. the bug isn’t in the slow query. it’s in everything around it.</p>

<p><em>(side note: this is why <code>pg_stat_user_tables.n_dead_tup</code> is the number i look at first, not query duration. by the time query duration spikes, the dead tuples already told you what was coming.)</em></p>

<h2 id="the-walk-in-cooler">the walk-in cooler</h2>

<p>this is where it gets worse and where the analogy earns its keep, because i want you to understand <code>DataFileRead</code> without making you read the postgres source.</p>

<p><code>shared_buffers</code> is the prep counter—the small workspace next to the stove where cooks keep the ingredients they’re using <em>right now</em>. it’s fast because everything is within arm’s reach. the walk-in cooler in the back is disk. ingredients in the cooler are slower to grab because somebody has to walk back there, open the door, find the thing, walk it back to the line.</p>

<p>in a healthy kitchen, almost everything a cook needs is already on the prep counter. they grab it and go. fast.</p>

<p>now bloat shows up. every shelf in the walk-in has 60x more stuff on it, and most of it is rotten food nobody’s allowed to throw out because the busser is locked out (probably again, since you’re reading this). the prep counter is still the same size. it doesn’t fit the working set anymore. cooks who used to find their ingredients on the counter now have to walk to the walk-in for every order, dig through piles of rotten food to find the one good onion, and walk it back.</p>

<p>they’re not cooking. they’re walking. and standing in front of the cooler with the door open.</p>

<p><code>DataFileRead</code> is pg telling you “this cook is currently standing in front of the walk-in waiting for an ingredient.” you can see it in <code>pg_stat_activity</code>—there’s a column called <code>wait_event</code>, and when it says <code>DataFileRead</code>, that backend is not doing work, it’s waiting on the kernel to hand it a page from disk. in a healthy db you barely see this column populated. in a bloated one, every cook in the kitchen is stuck on it. you open <code>pg_stat_activity</code> during the incident and the <code>wait_event</code> column is just <code>DataFileRead</code>, <code>DataFileRead</code>, <code>DataFileRead</code> all the way down.</p>

<p>turtles.</p>

<p>the buffer cache hit ratio is the metric for “what percent of the time did the cook find what they needed on the prep counter vs having to walk to the cooler.” healthy kitchen, 99%. bloated kitchen, less than 93%. the difference between those two numbers sounds small. it isn’t, watch: 99% uptime vs 93% uptime—your business is dead.</p>

<p>grabbing something off the prep counter is ~100 nanoseconds. walking to the walk-in is ~100 microseconds. that’s 1000x slower. drop the hit ratio from 99% to 60% and your average query gets dramatically slower without anything in the app or the query plan changing at all.</p>

<p><em>(the query plan is just pg’s recipe for how to cook the order. same recipe. same ingredients. same steps. just way more walking.)</em></p>

<p>and this is the part that makes people lose their minds during the incident: nothing changed. no deploy. no new code. wow, weird, nothing changed! the recipe must still be the same because the indexes are the same and my query certainly didn’t change because it was always <code>select * from users</code>.</p>

<p>but your graphs are vertical and everyone is on a call. and the answer is: the heap got fat (more dead rows piled up on the shelves), the prep counter didn’t, and now every cook is walking to the cooler for every order. the only thing that changed is the busser stopped being allowed to throw out rotten food, because of the 3-hour query nobody noticed.</p>

<p><code>DataFileRead</code> in <code>pg_stat_activity.wait_event</code> is the smoking gun. it’s pg pointing at the cooler door and saying “this is where your latency went.”</p>

<h2 id="one-slow-table-blocks-the-whole-restaurant">one slow table blocks the whole restaurant</h2>

<p>if someone’s slowly eating at table 7 (long <code>SELECT</code> holding <code>AccessShareLock</code>) and a contractor shows up to repaint table 7 (<code>ALTER TABLE</code> wanting <code>AccessExclusiveLock</code>), the contractor waits.</p>

<p>and then every new customer who wants table 7 lines up behind the contractor—even the ones who’d be done in 2 minutes—because pg’s lock queue is fifo. doesn’t matter that a fresh <code>SELECT</code> only needs <code>AccessShareLock</code> and would be compatible with the running one; it’s behind the <code>ALTER</code>, so it waits.</p>

<p>one slow eater freezes the whole section.</p>

<p>translated: one slow query blocks a migration, and the migration blocks every other query on that table. deploys hang. writes hang. and the worst part is you’ll see the <em>waiting</em> queries in <code>pg_stat_activity</code> and think <em>those</em> are the problem, when really it’s the 3-hour <code>SELECT</code> at the front of the queue.</p>

<h2 id="spilled-food-keeps-spilling">spilled food keeps spilling</h2>

<p>when a query needs more memory than <code>work_mem</code> allows, it spills to disk—hash joins, sorts, ctes, the usual suspects. those temp files live in <code>base/pgsql_tmp</code> on the data volume.</p>

<p>right now those queries get cut off before they finish spilling. without the timeout they finish. and a single bad join can write tens of gigs.</p>

<p>we’ve already had an outage this year from running out of floor space. this is a direct path to another one. and <code>pg_stat_tmp</code> doesn’t alert on “about to fill the disk,” it just fills it. the next thing that happens is pg can’t write wal, replication breaks, and now you’re in a real outage instead of a slow one.</p>

<h2 id="youre-firing-the-smoke-alarm-because-it-keeps-going-off">you’re firing the smoke alarm because it keeps going off</h2>

<p>right now slow queries show up in sentry as <code>PG::QueryCanceled</code> with a stack trace pointing at the exact line of code. that’s free debugging info—pg is literally telling you which activerecord call is the problem.</p>

<p>turn off the timeout and the alerts disappear because there’s no cancellation to raise. the queries don’t get faster. you just stop seeing them.</p>

<p>and the symptom isn’t gonna be “the bad query is slow,” it’s gonna be “everything is slow,” because the bloat pileup and the cache evictions affect every query touching those tables. way harder to debug at 3am from <code>pg_stat_activity</code> than from a stack trace pointing at the controller action.</p>
]]></content>
    
    <summary>so you got tired of alerts that your queries were timing out. that’s bad because alerts are good even though nobody likes the alerts. my mom didn’t like acknowledging she might have cancer either so she just ignored the headaches and vertigo and balance issues and then it was like ope stage IV brain cancer.
</summary>
    
    
    <category term="engineering"/>
    
  </entry>
  
  <entry>
    <title>We made this hard: over-engineering the web</title>
    <link href="/posts/we-made-this-hard/" rel="alternate" type="text/html"/>
    <id>/posts/we-made-this-hard/</id>
    <published>2026-04-28T00:00:00+00:00</published>
    <updated>2026-04-28T00:00:00+00:00</updated>
    <content type="html"><![CDATA[<h1 id="we-made-this-hard">We made this hard</h1>

<p>The job hasn’t changed in thirty years. A request comes in. You return some HTML. The browser draws it. Maybe you sprinkle some CSS on top so it doesn’t look like a 1996 GeoCities fever dream, or maybe you do so it does. That’s the whole gig.</p>

<p>Somewhere along the way we convinced ourselves this hard.</p>

<p>It isn’t. It really, really isn’t. And the people paying for our confusion are the ones who can least afford it: the restaurant owner who just wants a menu online, the babysitter trying to put up a contact form, and—maybe most painfully—the indie dev who’s been “almost ready to launch” for two years because they’re still picking between four different ways to host Postgres, which came to mind only having a nightmare post-engineering-blog post they saw on HackerNews.</p>

<p>I want to talk to both of those groups. This isn’t a slight against React or Kubernetes or any other tool that exists for a reason for companies in the 99th percentile of scale. It’s a screed against using them as the <em>default</em>. There’s a difference between “we needed this because we have a real problem” and “we used this because the FAANG blog post said they did.”</p>

<p>And their problems are your problems, just as my problems are your problems, right?</p>

<h2 id="i-had-a-website-in-2004">I had a website in 2004</h2>

<p>I was ten. It ran on a 100MB shared hosting plan that cost something like four bucks a month. PHP, MySQL, cPanel when it was cool, and an unreasonable amount of confidence because I knew how to run a lemonade stand. It served real traffic! It didn’t fall over! I didn’t know what a load balancer was and it didn’t matter whatsoever, because nobody on earth needed me to have one.</p>

<p>The 2004 version of today’s $4/month would have been a small datacenter, just to serve HTML.</p>

<p>The hardware got absurdly, stupidly better. Disks got faster by something like four orders of magnitude. RAM got cheap (and then expensive so we can make squirrels with horns). CPUs got many. SQLite, running on a laptop, will outperform what most Fortune 500 companies considered “the database tier” in 2008.</p>

<p>And yet shipping a website is harder than it was when I was ten. That makes sense.</p>

<h2 id="what-actually-changed-spoiler-not-much">What actually changed (spoiler: not much)</h2>

<p>Browsers got better. CSS got <em>way</em> better—grid and flexbox alone fixed about eighty percent of what used to require a table-based layout and tears (tables: still dope). HTTPS became <em>free</em>, easy, and automatic. HTTP/2 quietly fixed the things people used to pre-optimize around. Static site generators became a thing so we don’t have partials to include headers and footers. Postgres got good enough to be the only database most people will ever need. SQLite got fast enough that “just use SQLite” is, increasingly, the correct answer.</p>

<p>Almost every meaningful platform change in the last twenty years made things <em>easier</em>. The complexity isn’t coming from the platform.</p>

<p>It’s coming from us.</p>

<h2 id="webs-complexity-industrial-complex">Web’s complexity industrial complex</h2>

<p>Let me walk me through how a small business gets a website in 2026, according to an aspiring entrepreneur who makes widgets, and has a friend who is a junior software engineer at a vendor for Intuit.</p>

<p>WidgetCo person hires InuitVendorJuniorStar, “hey I need something.” “Sure I can do it! Wait what is it?”</p>

<p>Before even getting started, InuitVendorJuniorStar pulls up a Twitter thread they bookmarked because they saw someone was successful, and they’re certain that copying that recipe is how they too can be successful. That thread mentioned Next.js or Remix or whatever framework was hot four months ago and will be embarrassing eighteen months from now. They set up a build pipeline. They add TypeScript. They add ESLint, Prettier, Husky, lint-staged, a commit message linter because god forbid the commit messages be inconsistent. They deploy to Vercel because that’s what the tutorial said. They add a CDN because performance. They add analytics from three vendors. They add a cookie banner because of the analytics. They add a CMS because the client might want to edit text someday. They add a headless CMS because WordPress is so yesterday.</p>

<p>WidgetCo’s site is a restaurant menu. A fucking menu.</p>

<p>The build outputs more JavaScript than the entire site needs to function (it really just needs a hamburger menu), ships hydration logic for a page with zero interactivity, and takes six seconds to load (but it has an animation spinner!) on the kind of phone the actual customer is putting in their pocket. The original brief may as well have been: show people what’s on today’s menu, and add a static Google Maps embed.</p>

<p>The now-half-baked result (note: originally I put product here but changed to result) is a single-page application that requires a service worker to display the address, and it won’t give you directions unless you give it your <code>navigation.location</code> permissions.</p>

<p>This isn’t an exaggeration. I’ve seen it. You’ve seen it. We’ve all seen it. The babysitter’s site has a CDN because her boyfriend heard about one on Discord. The four-location pizza place has Kubernetes—I am not making this up, I have personally seen a pizza place running their WordPress instance with a link to Toast’s ordering system on Kubernetes—and somewhere a cloud provider’s quarterly numbers went up by less than a rounding error.</p>

<p>The patterns we cargo-culted from Netflix and Stripe and Facebook and Google and Amazon don’t apply. Netflix has a different problem. Stripe has a different problem.</p>

<p>The problem was “people should be able to find our phone number and hopefully find us on Google.”</p>

<p>That problem was solved in 1996.</p>

<h2 id="the-indie-dev-trap">The indie dev trap</h2>

<p>Now the other audience:</p>

<p>If you’re an indie dev or a solo founder, your enemy isn’t Kubernetes. Your enemy is subtler. Your enemy is the version of you that opens a new tab and starts reading “Postgres on RDS vs. Supabase vs. Neon vs. self-hosted vs. Planetscale vs. Turso vs. hey-I-launched-this-this-morning-at-11am-on-a-Tuesday-along-side-everything-else-that-launches-at-11am-on-a-Tuesday.</p>

<p>Your enemy is the version of you that’s been “architecting” for six weeks and hasn’t had a single production user.</p>

<p>Here’s the pattern. You have an idea. You get excited. You start a project. And then, instead of building (or continuing to build) the thing, you start optimizing for a scale you do not have and will not have for a long time, possibly ever. You pick a database based on what <em>might</em> happen if you got hugged to death by Hacker News <em>times fifty</em>. You set up a queue for operations that take forty milliseconds. You build a microservice architecture for an app with three endpoints. You spend a week on the deploy pipeline. Then another week on the <em>staging</em> deploy pipeline. Then a weekend on the observability stack, because how would you know if it broke?</p>

<p>It hasn’t broken. In fact it hasn’t done anything  because it hasn’t been used by a single human being other than you and maybe your mom. There is nothing to observe, and you don’t even know what to measure.</p>

<p>Pre-optimization feels like you’re doing something so you do it. It even feels like progress. You’re typing and asking reddit. You’re committing and solving problems. You’re learning new tools because you’re being a Real Engineer.</p>

<p>You ain’t doing any productive shit.</p>

<p>You are procrastinating in the most expensive possible way, because if you’re still architecting, you can’t be judged yet. And you’re probably afraid to be judged, because heaven forbid an early adopter finds a bug <em>and tells you for free and continues to use your product anyway because it solves their problem</em>. Procrastinating means nobody can tell you the idea was bad, nobody can tell you nobody wanted it (because you totally built without a customer). You’re safe in the architecture diagram. You’re safe in the stack-shopping loop. The moment you ship, the safety ends and you find out whether you built something anyone cares about.</p>

<p>That’s the real fear. The architecture is just where it hides.</p>

<p>I’ve watched people I like avoid shipping because they couldn’t decide on the right framework (even though they’ve mastered one!), the right database (for their boring-ass ACID transactions!), the right hosting setup (because they thought they needed high-availability off the rip!). The thing that would’ve taken them a weekend to put up on a $4 VPS turned into a year-long Ferret Dress-Up Championship in which the prize is also a ferret.</p>

<p>For whatever reason we refuse to accept that the scale problems we’re worrying about are <em>good problems</em>. They mean people use your fucking thing. You do not have those problems, and you probably won’t because you’re too busy with eighty tabs open comparing providers and reading anecdotes from people you don’t even know who probably thought they were smart by writing down <em>what I learned</em> when in fact they have the same paralysis as you do, or they have demand-driven scale problems and have solved it and now have time to write about it.</p>

<p>Nobody writes about it in the middle of it.</p>

<p>When you do have these scaling problems, you will have users (and hopefully revenue), you will have signal, you will have measure, and you will have time to fix them. And the fix will almost always be embarrassingly boring: add an index, throw money at a bigger box, change your cache key.</p>

<p>It will not be “I should’ve started with Kafka and k8s.”</p>

<p>You can scale a boring monolith on one VPS further than you think. A modest Rails or Django or Phoenix app on a single 4-core box serves tens of thousands of users without sweating (my boring Rails app served at least 33,004 paying users according to the government, and it was not simple). Stack Overflow was lean as hell for years when they mattered. The thing you’re scared of is rarer than you think and easier to solve than you imagine.</p>

<h2 id="marketing-people-marketing-fear">Marketing people marketing fear</h2>

<p>The marketing teams are the real winners here.</p>

<p>Not the users of the product they market. The users who want pages that load. Not the small business owner—they want their phone to ring. Not the indie dev who hasn’t shipped because they’re still picking a stack.</p>

<p>Cloud providers benefit. Framework maintainers benefit from popularity. The conference circuit benefits. Engineers writing the complexity benefit—it’s more interesting work, it pays better, it looks better on a resume than “I deployed a Rails monolith to a single server and it’s been up for three years.”</p>

<p>I’m not saying any of this is a conspiracy or fraud. It’s not. But it’s an incentive structure, and the incentive structure says: build the technically impressive thing, talk about the technically impressive thing, hire people who can build the technically impressive thing.</p>

<p>The babysitter does not have a seat at that table. Neither does the indie dev who hasn’t shipped.</p>

<p>It’s worth saying out loud anyway. The reason we keep recommending complicated stacks isn’t that they’re correct for the situation. It’s that they’re correct for <em>us</em>.</p>

<h2 id="what-you-probably-actually-need">What you probably actually need</h2>

<p>Your taco spot needs cheap shared hosting so you can install WordPress with a click; maybe buy a theme from ThemeForest. If they need a email at their domain (they may sure as well be fine with a Gmail account) most hosting providers have Google Workspace addons so you can manage everything under one roof so they don’t get stuck using an IP with poor reputation by a company whose specialty is not email. So yes, $15/month or whatever; or you can go use Yandex or Zoho or whatever else.</p>

<p>For ninety-nine percent of websites and six-nines percent of pre-launch indie projects, the answer is humiliatingly simple:</p>

<ul>
  <li>A $6/month VPS or a small computer under your desk—seriously, a Mac mini under your desk handles a lot</li>
  <li>Static HTML for the babysitter, or an app in whatever language you already know (Ruby, Python, PHP, Go, Elixir, doesn’t matter)</li>
  <li>Postgres or SQLite on the same machine as the app, with periodic backups to some probably-S3-compatible storage and a well-tested script to restore that database. 100GB restores in like 10 minutes and you can probably stomach 10 minutes of downtime</li>
  <li>Nginx or Caddy in front</li>
  <li>Cloudflare’s free tier if you want a “CDN”, which—sure, why not, it’s free</li>
</ul>

<p>That’s it. That stack will serve more traffic than you will get. It will be up at 3am save for the datacenter burning down. It will not page you. It will cost less than your coffee. You can build it in a weekend and forget about it for two years.</p>

<p>If you’re an indie dev: ship on this. Get users. Find out if anyone wants the thing. <em>Then</em> worry about the rest.</p>

<h2 id="the-real-cases-briefly">The real cases, briefly</h2>

<p>There are real reasons to use the complicated stuff. Genuine multi-region latency requirements. Regulatory constraints that demand specific architectures. Team sizes where the coordination overhead of a monolith breaks down. Actual scale—the kind where you have so many users that the simple answer stops working.</p>

<p>These are real and they are rare. If you have to wonder whether you’re in this category, you aren’t.</p>

<p>And here’s the part that should be liberating: boring stacks don’t trap you. A Rails app on a single server is the easiest thing in the world to migrate out of, because it’s standard, because it’s understood, because every engineer you’ll ever hire has worked on one. The lock-in stories are about <em>complex</em> setups, not simple ones. You can always add complexity later. It’s much, much harder to remove it.</p>

<h2 id="the-job">The job</h2>

<p>A request comes in. You return some HTML. The browser draws it.</p>

<p>That’s the web. That’s been the web since 1995. Everything we’ve built on top is optional. Some of it’s even useful, but almost none of it is necessary for what most of us are actually building, and a striking amount of it is being deployed in situations where it makes things measurably worse.</p>

<p>Build the smallest thing that works. Ship it. The scale problems are the good ones—and you don’t have them yet.</p>
]]></content>
    
    <summary>We made this hard
</summary>
    
    
    <category term="engineering"/>
    
  </entry>
  
  <entry>
    <title>How Postgres works at scale</title>
    <link href="/posts/how-postgres-works-at-scale/" rel="alternate" type="text/html"/>
    <id>/posts/how-postgres-works-at-scale/</id>
    <published>2026-04-11T00:00:00+00:00</published>
    <updated>2026-04-11T00:00:00+00:00</updated>
    <content type="html"><![CDATA[<p>A single idle-in-transaction connection can starve every table in your database. Most senior Rails devs don’t know that.</p>

<p>You learned MVCC the way most of us did—”writers don’t block readers,” concurrent transactions get isolated snapshots, UPDATE doesn’t lock the table. That’s marketing copy. The operational reality is that those snapshots have a cost, and the cost compounds across every table at once if one connection gets stuck.</p>

<p>I’m going to try to translate what this mental model is that I’ve built while building and operating a Rails-based data pipeline that handles 10TB/day. Not “what is MVCC”—you’ve heard that—but instead consequences of MVCC. You know, the ones that turn a quiet Monday into a panic akin to “why is production at 95% disk.”</p>

<p>This application of mine hangs out on a single server.</p>

<h2 id="the-application">The application</h2>

<p>Before you’re like “omg Rails” I will disclose that ActiveRecord is only used at the read layer for the user-facing API, for its database connection, and the orchestration of the pipeline. The work is largely done in SQL.</p>

<p>I have a nice little helper that I’ve enjoyed using. You may enjoy it too; here are the good parts:</p>

<pre><code class="language-ruby">module CoolBeans
  class SQL
    class Error &lt; StandardError; end
    class UnknownProfile &lt; Error; end
    class AlreadyInTransaction &lt; Error; end
    class InvalidSetting &lt; Error; end

    class &lt;&lt; self
      def execute(profile, connection: ActiveRecord::Base.connection, &amp;block)
        raise ArgumentError, "block required" unless block

        new(profile: profile, connection: connection).call(&amp;block)
      end

      def settings_for(profile)
        config.fetch(profile.to_s) do
          raise UnknownProfile, "no profile=#{profile} in config/sql.yml for env=#{Rails.env}"
        end
      end

      def profiles
        config.keys
      end

      def reload!
        @config = nil
      end

      def config
        @config ||= Rails.application.config_for(:sql).to_h.deep_stringify_keys
      end
    end

    def initialize(profile:, connection:)
      @profile = profile.to_s
      @connection = connection
      @settings = self.class.settings_for(@profile)
    end

    def call
      if connection.transaction_open?
        raise AlreadyInTransaction,
              "SQL.execute(#{profile.inspect}) cannot nest inside an existing transaction; " \
                "SET LOCAL would leak past this block. Call it at the top of the unit of work."
      end

      logger = ActiveRecord::Base.logger
      original_level = logger&amp;.local_level

      begin
        if logger
          logger.silence(Logger::ERROR) do
            connection.transaction do
              apply_settings!
              logger.silence(original_level) do
                yield(connection)
              end
            end
          end
        else
          connection.transaction do
            apply_settings!
            yield(connection)
          end
        end
      ensure
        ActiveSupport::Notifications.unsubscribe(subscriber) if subscriber
        warn_if_insert_heavy(stats)
      end
    end

    private

    attr_reader :profile, :connection, :settings

    def apply_settings!
      settings.each do |key, value|
        connection.execute("SET LOCAL #{key} = #{connection.quote(value.to_s)}")
      end
    end

    module ConnectionMixin
      def tuned(profile, &amp;block)
        proxy = Proxy.new(self, profile)
        return proxy unless block

        proxy.call(&amp;block)
      end
    end

    class Proxy
      DELEGATED = %i[
      execute exec_query exec_insert exec_update exec_delete
      select_all select_one select_value select_values
    ].freeze

      def initialize(connection, profile)
        @connection = connection
        @profile = profile
      end

      def call(&amp;block)
        SQL.execute(@profile, connection: @connection, &amp;block)
      end

      DELEGATED.each do |m|
        define_method(m) do |*args, **kwargs, &amp;blk|
          SQL.execute(@profile, connection: @connection) do |c|
            c.public_send(m, *args, **kwargs, &amp;blk)
          end
        end
      end
    end
  end
end
</code></pre>

<p>That fancy config file:</p>

<pre><code class="language-yaml">development:
  bulk_import:
    statement_timeout: 0
    lock_timeout: 0
    synchronous_commit: "off"
    maintenance_work_mem: "16GB"
    work_mem: "512MB"
    max_parallel_maintenance_workers: 8
    max_parallel_workers_per_gather: 8
  promote:
    statement_timeout: 0
    lock_timeout: 0
    synchronous_commit: "off"
    session_replication_role: "replica"
    maintenance_work_mem: "16GB"
    work_mem: "512MB"
    max_parallel_maintenance_workers: 8
    max_parallel_workers_per_gather: 8
  create:
    statement_timeout: 0
    lock_timeout: 0
    synchronous_commit: "off"
    session_replication_role: "replica"
    maintenance_work_mem: "16GB"
    work_mem: "32GB"
    max_parallel_maintenance_workers: 8
    max_parallel_workers_per_gather: 8

production:
  bulk_import:
    statement_timeout: 0
    lock_timeout: 0
    synchronous_commit: "off"
    maintenance_work_mem: "16GB"
    work_mem: "512MB"
    max_parallel_maintenance_workers: 8
    max_parallel_workers_per_gather: 8
  promote:
    statement_timeout: 0
    lock_timeout: 0
    synchronous_commit: "off"
    session_replication_role: "replica"
    maintenance_work_mem: "16GB"
    work_mem: "512MB"
    max_parallel_maintenance_workers: 8
    max_parallel_workers_per_gather: 8
  create:
    statement_timeout: 0
    lock_timeout: 0
    synchronous_commit: "off"
    session_replication_role: "replica"
    maintenance_work_mem: "16GB"
    work_mem: "32GB"
    max_parallel_maintenance_workers: 8
    max_parallel_workers_per_gather: 8
</code></pre>

<h2 id="before-we-get-into-things">Before we get into things</h2>

<p>A lot of this becomes irrelevant if your scale is not scale.</p>

<h2 id="mvcc-compressed">MVCC, compressed</h2>

<p>Every row has two hidden columns: <code>xmin</code> (the transaction that wrote it) and <code>xmax</code> (the transaction that deleted or superseded it). A transaction reads a row when <code>xmin</code> is committed and visible to its snapshot, and <code>xmax</code> either isn’t set or isn’t visible. That’s how postgres avoids locking readers against writers—they’re looking at different versions of the same rows, decided by visibility rules instead of locks.</p>

<p>UPDATE doesn’t modify a row in place. It writes a new tuple, sets the old one’s <code>xmax</code>, and leaves the old version on the page until vacuum proves nobody can still see it. DELETE does even less—it sets <code>xmax</code> and walks away. The space stays allocated. Vacuum is the garbage collector. Without it, the heap grows monotonically and the indexes follow.</p>

<p>That’s the model. The interesting question is what happens when vacuum can’t or won’t do its job.</p>

<h2 id="the-xmin-horizon-and-cluster-wide-vacuum-suckage">The xmin horizon and cluster-wide vacuum suckage</h2>

<p>Vacuum can only reclaim a dead tuple if it’s invisible to every active snapshot in the database. Not just transactions touching this table—every transaction in the cluster. Postgres tracks the oldest xmin still in use as the xmin horizon, and refuses to remove any tuple newer than that horizon.</p>

<p>So: one transaction starts at 9am, holds the snapshot, and goes idle. At 9:01, a thousand UPDATEs land on <code>users</code>. At 9:02, ten thousand UPDATEs land on <code>events</code>. Vacuum runs all afternoon, looks at every dead tuple in every table, and skips them. They’re newer than the 9am horizon.</p>

<p>Every table bloats. Indexes bloat worse. Replicas slow because they’re replaying the WAL backlog of those updates and can’t trim WAL until the primary tells them it’s safe. Disk fills. Plans degrade. The query that was 8ms last week is 400ms now and the indexes look fine in <code>\d</code>.</p>

<p>The cause is usually one of:</p>

<ul>
  <li>A Rails console someone left open with a <code>transaction do</code> and a <code>binding.pry</code> on the stack</li>
  <li>A <code>pg_dump</code> running unexpectedly long against the primary</li>
  <li>A worker that began a transaction, raised before commit, and ended up in a connection state where the rollback never landed</li>
  <li>A long analytics query someone runs from a notebook against production</li>
</ul>

<p>The diagnostic—the one query you should be able to type from memory in an outage:</p>

<pre><code class="language-sql">SELECT pid, now() - xact_start AS duration, state, query
FROM pg_stat_activity
WHERE state = 'idle in transaction'
ORDER BY duration DESC;
</code></pre>

<p>If anything shows up with a duration measured in hours, that connection is your xmin horizon, and your bloat is its fault. The fix is <code>pg_terminate_backend(pid)</code>. The follow-up is <code>idle_in_transaction_session_timeout</code> set to something sane—five or ten minutes—so the database protects itself when you forget.</p>

<p>This is the most underused safety net in postgres. The default is zero. Zero means “let one developer’s rails console eat the cluster.”</p>

<h2 id="wal-checkpoints-and-the-bulk-import-storm">WAL, checkpoints, and the bulk-import storm</h2>

<p>Every write goes to the write-ahead log first. The actual data pages get dirtied in shared buffers and written to disk later, at checkpoint time. WAL is sequential and fast. Random heap writes are not. That’s the trick that lets postgres survive a crash and also keeps writes cheap.</p>

<p>Checkpoints flush all dirty buffers to disk so WAL up to that point can be discarded. They run on a schedule (<code>checkpoint_timeout</code>) or when WAL volume hits <code>max_wal_size</code>, whichever comes first. The schedule is gentle. The volume trigger is not.</p>

<p>Here’s the failure mode: You start a bulk import; a worker is COPYing into staging tables and you’re generating WAL faster than the schedule can absorb. You hit <code>max_wal_size</code> halfway between scheduled checkpoints, postgres triggers a forced checkpoint, IO saturates, and every query holding a buffer pin waits its turn behind the flush. Latency on totally unrelated requests spikes for ten or twenty seconds.</p>

<p>Pagers scream.</p>

<p>The fix is configuration, not architecture:</p>

<ul>
  <li><code>max_wal_size</code> big enough that your peak write rate doesn’t hit it between scheduled checkpoints. I use 48GB. That’s not a typo—modern disks are cheap, and the alternative is checkpoint storms.</li>
  <li><code>checkpoint_timeout</code> long enough that the schedule does the work. I use 30 minutes.</li>
  <li><code>checkpoint_completion_target = 0.9</code> so postgres spreads the flush across 90% of the interval instead of bunching it.</li>
</ul>

<p>You’ll see the symptom in <code>pg_stat_bgwriter</code>: <code>checkpoints_req</code> (forced) climbing while <code>checkpoints_timed</code> (scheduled) stays flat. If <code>checkpoints_req</code> is more than a small fraction of <code>checkpoints_timed</code>, your <code>max_wal_size</code> is too small.</p>

<p>The thing nobody tells you: bigger <code>max_wal_size</code> is also disk space. 64GB of WAL means 64GB of disk reserved for WAL. Budget accordingly.</p>

<h2 id="the-lock-graph-that-actually-matters">The lock graph that actually matters</h2>

<p>Postgres has eight lock modes. You don’t need to memorize the table. You need to know:</p>

<ul>
  <li><code>AccessExclusive</code> blocks everything, including reads. This is what <code>ALTER TABLE</code>, <code>DROP TABLE</code>, and most non-<code>CONCURRENTLY</code> index operations take. If you take it on a hot table during peak traffic, your app stops. You may have done this.</li>
  <li><code>ShareUpdateExclusive</code> is what <code>VACUUM</code>, <code>ANALYZE</code>, and <code>CREATE INDEX CONCURRENTLY</code> take. It conflicts with itself. You cannot run two <code>VACUUM</code> on the same table at once. Autovacuum can be blocked by your manual <code>VACUUM</code>, and vice versa.</li>
  <li><code>RowExclusive</code> is what <code>UPDATE</code>, <code>DELETE</code>, <code>INSERT</code> take. It conflicts with locks meant for schema changes, but not with itself. That’s why your app keeps writing while another connection is also writing.</li>
</ul>

<p>That’s most of what matters in production. The other modes exist mainly because schema changes need to coordinate with reads in specific ways. The two consequential rules: <code>AccessExclusive</code> stops the world, and <code>ShareUpdateExclusive</code> serializes maintenance with itself.</p>

<p>The Rails-shaped trap: a migration that runs <code>add_column</code> with a default value on a table with millions of rows, in a transaction. <code>AccessExclusive</code> for the duration of the rewrite. Every read on the table queues. PG11+ made the constant-default case cheap—adding a column with a literal default no longer rewrites the table — but <code>ALTER COLUMN TYPE</code>, adding <code>NOT NULL</code> the wrong way, and indexes without <code>CONCURRENTLY</code> all still rewrite.</p>

<h2 id="the-blocker-tree">The blocker tree</h2>

<p>When the app feels stuck—requests piling up, response times climbing, no obvious culprit in the application logs—the question is “who’s blocking who.” Postgres ships the answer.</p>

<pre><code class="language-sql">SELECT
  blocked.pid AS blocked_pid,
  blocked.query AS blocked_query,
  blocking.pid AS blocking_pid,
  blocking.query AS blocking_query,
  now() - blocking.xact_start AS blocking_duration
FROM pg_stat_activity blocked
JOIN pg_stat_activity blocking
  ON blocking.pid = ANY (pg_blocking_pids(blocked.pid))
WHERE blocked.wait_event_type = 'Lock';
</code></pre>

<p><code>pg_blocking_pids</code> returns the array of pids that are blocking a given pid. The join expands the graph. In practice the output is short—usually one or two blockers at the root, with a fan of blocked queries waiting on them. The root is what you kill.</p>

<p>I keep this in a snippet file. So should you. The day you need it, you don’t have time to look it up.</p>

<p>What you do with the result is the harder question. <code>pg_cancel_backend(pid)</code> is polite—it asks the query to stop. <code>pg_terminate_backend(pid)</code> is not—it kills the connection and rolls the transaction back. Polite first, terminate second. If the polite kill doesn’t take inside a few seconds, the query is in an uninterruptible kernel call (usually IO) and you need the hammer.</p>

<h2 id="toast-and-the-silently-slow-big-column-problem">TOAST and the silently-slow big-column problem</h2>

<p>Postgres pages are 8KB. A row larger than the page can’t fit. TOAST—The Oversized Attribute Storage Technique (postgres devs have a sense of humor)—handles this by compressing large values, breaking them into chunks, and storing the chunks in a side table. The main row gets a pointer.</p>

<p>The compression and chunking are transparent. The performance characteristics are not.</p>

<p>A <code>text</code> or <code>jsonb</code> column that grows from kilobytes to megabytes—say, a vendor response payload you started caching—quietly turns every SELECT against the row into an extra round of TOAST chunk fetches. The query plan looks identical. EXPLAIN shows the same Index Scan. The actual IO doubles or triples.</p>

<p>The symptoms:</p>

<ul>
  <li>Bitmap Heap Scan reading way more buffers than rows</li>
  <li>A query that was fast last quarter is slow now and nothing about it changed</li>
  <li>Disk read volume on the table is wildly disproportionate to row count</li>
</ul>

<p>The fixes, in order of effort:</p>

<ul>
  <li><code>ALTER TABLE x ALTER COLUMN big_jsonb SET STORAGE EXTERNAL</code> to skip compression if it isn’t helping</li>
  <li><code>ALTER TABLE x ALTER COLUMN big_jsonb SET COMPRESSION lz4</code> (PG14+, I think?) for faster decompression</li>
  <li>Stop selecting the column when you don’t need it. If your AR model has a <code>payload</code> column you only need on detail pages, set <code>self.ignored_columns = %w[payload]</code> on a slim subclass for list queries. Or use <code>select(:id, :name, ...)</code> everywhere it matters.</li>
  <li>Move the column to a side table you join in only when needed. Sounds redundant—postgres already moved it—but the catalog lookup, planner cost, and chunked decompression are real overhead avoided when the column isn’t selected at all.</li>
</ul>

<p>The deeper lesson: a <code>jsonb</code> column is not free, and the cost is hidden under a layer that doesn’t show up in <code>EXPLAIN</code> until you read it carefully.</p>

<h2 id="transaction-id-wraparound">Transaction ID wraparound</h2>

<p>Every transaction gets a 32-bit ID. Postgres needs <code>xmin</code> to determine row visibility. 32 bits gives you about 4 billion transactions, after which the counter wraps. To keep visibility correct across the wrap, postgres “freezes” old rows—marks them as definitively visible to everyone—so their original <code>xmin</code> doesn’t matter anymore. Vacuum does the freezing.</p>

<p>If autovacuum can’t keep up—usually because it’s blocked, throttled, or starving—the database approaches wraparound. At a configurable threshold postgres starts running aggressive anti-wraparound vacuums. If those can’t keep up, postgres shuts down the database to prevent corruption. This has happened to enough big companies to be a meme.</p>

<p>The query you should run on a schedule:</p>

<pre><code class="language-sql">SELECT datname, age(datfrozenxid)
FROM pg_database
ORDER BY 2 DESC;
</code></pre>

<p>Default <code>autovacuum_freeze_max_age</code> is 200 million. A reasonable alarm is 500 million. Panic at 1.5 billion. If you’re seeing 1.5B and rising, something has been blocking autovacuum for a long time—usually that idle-in-transaction connection from earlier, or a manual VACUUM holding ShareUpdateExclusive on a huge table, or autovacuum workers maxed out on something else.</p>

<p>This is one of those operational concerns Rails developers never see until they own a database. Then they see it once and never forget.</p>

<h2 id="the-catalog-as-data">The catalog as data</h2>

<p>For whatever reason we don’t believe that the postgres catalog is not metadata. It’s tables. Real tables. With rows. You can query them. It’s like the postgres devs are smart.</p>

<p><code>pg_class</code> is one row per table, index, sequence, or other relation. <code>pg_attribute</code> is one row per column, across every relation. <code>pg_index</code> is one row per index with the column references and the predicate for partials. <code>pg_proc</code> is one row per function and stored procedure.</p>

<p>These aren’t an API surface that postgres “exposes”—they’re how postgres stores its own bookkeeping, and you can SELECT from them like anything else. You can JOIN them. You can build views over them. You can point an ActiveRecord model at them.</p>

<p>A pattern from a codebase I work in:</p>

<pre><code class="language-ruby">module StoredProcedures
  class BaseStoredProcedure &lt; ApplicationRecord
    self.table_name = "pg_proc"
    self.primary_key = "oid"
  end
end
</code></pre>

<p><code>StoredProcedures::BaseStoredProcedure.where("proname LIKE 'tasker_%'")</code> returns every stored procedure prefixed with <code>tasker_</code> in the catalog. AR doesn’t care that the table happens to be the function catalog—it’s a table with columns, AR is happy.</p>

<p>(I use this to manage migrations for stored procedures, which I’m usually against but at the scale of this application, I love saving 10ms across inserts because I have a couple million to do a day.)</p>

<p>What this enables, once you accept the framing:</p>

<ul>
  <li>A migration that asserts a stored procedure exists with the right signature, written as an AR query</li>
  <li>A test that fails when a function definition drifts from what your code expects</li>
  <li>A diagnostic page in your admin tool that lists every function with its source, fed by <code>pg_get_functiondef(oid)</code></li>
  <li>A check-on-boot that loads function definitions from disk and compares them to the catalog, surfacing manual edits</li>
</ul>

<p>The framing scales further. A <code>StagingTable</code> model in the same codebase points at <code>pg_stat_user_tables</code> for live row counts and last-vacuum timestamps. It’s an AR model whose underlying table is a postgres-maintained statistics view. The implementation is two lines. The capability is “show me the freshness of every staging table on the dashboard, with sorting and search, for free.”</p>

<p>Maybe I need to create some sort of gem that exposes these things.</p>

<p>Beginners don’t realize you can do this. Many seniors haven’t tried. It costs almost nothing to start, and it changes how you think about the database—the catalog isn’t a black box you query through <code>\d</code> or DBeaver. It’s data. Your application can read it and reason about it like anything else.</p>

<h2 id="what-this-buys-you">What this buys you</h2>

<p>The compressed version:</p>

<ul>
  <li>One idle transaction can bloat every table in the cluster. Set <code>idle_in_transaction_session_timeout</code>. Run the <code>pg_stat_activity</code> query when something feels off</li>
  <li>Bulk imports cause checkpoint storms. Bigger <code>max_wal_size</code>, longer <code>checkpoint_timeout</code>, completion target 0.9</li>
  <li><code>AccessExclusive</code> stops the world. Most migrations should not take it on hot tables during peak</li>
  <li>The blocker tree CTE is your “the app feels stuck” diagnostic. Memorize it or keep it in a snippet</li>
  <li>Big <code>jsonb</code> columns turn into TOAST overhead that doesn’t show in EXPLAIN. Stop selecting them when you don’t need them</li>
  <li>Wraparound is real. Monitor <code>age(datfrozenxid)</code>. Default thresholds will not save you if autovacuum is blocked</li>
  <li>The catalog is data</li>
</ul>

<p>None of this is exotic. It’s the operational side of postgres that the docs cover but the Rails ecosystem doesn’t talk about. Senior devs may have heard the words, but knowing what they cost in production is the difference.</p>
]]></content>
    
    <summary>A single idle-in-transaction connection can starve every table in your database. Most senior Rails devs don’t know that.
</summary>
    
    
    <category term="postgres"/>
    
  </entry>
  
  <entry>
    <title>Your README is a landing page, you just don&apos;t know it yet</title>
    <link href="/posts/landing-pages-for-open-source-maintainers/" rel="alternate" type="text/html"/>
    <id>/posts/landing-pages-for-open-source-maintainers/</id>
    <published>2026-03-24T00:00:00+00:00</published>
    <updated>2026-03-24T00:00:00+00:00</updated>
    <content type="html"><![CDATA[<p>I have a few open-source libraries that you could call popular. I use a lot of others.</p>

<p>Open source isn’t a README, but it’s the cheapest marketing a developer has.</p>

<p>A world-class marketing person opens your README the same way they’d open a landing page, because that’s what it is. If you just wrote it like internal documentation—like developers wane through all day—it’s going to become a sore. The marketing-brain version treats the README as a conversion funnel where the conversion is “developer tries this and tells a friend.”</p>

<p>You spent six months on the library and four minutes on the thing 99% of people will ever see of it. That math is bad.</p>

<p>I’m not a marketer, but I’ve studied enough about landing pages (and deployed them to production) to have some sort of opinion here. Is it good? I don’t know, it’s impractical (if not downright quirky) to A/B test.</p>

<h2 id="zero-in-on-your-hero">Zero in on your hero</h2>

<p>The first section before any scroll is the most valuable real estate in the entire project and almost everyone wastes it. It needs three things and not really anything else: a one-line positioning statement, a small cluster of badges that signal legitimacy to build trust with the reader (build, version, downloads, license), and optionally a logo or a single screenshot/gif if the thing is visual.</p>

<p>I left out “the project name” because the user already knows why they’re there. You can put that after this section.</p>

<p>Note: positioning statement, not description. There’s a difference. “X for Y” or “the Z that does W.” Not “a fast, flexible, feature-rich whatever.” “Redis is an in-memory data structure store” beats “Redis is a fast, flexible, feature-rich key-value system supporting…” every time and it’s not even close.</p>

<p>Caffeinate says <code>Caffeinate is a drip engine for managing, creating, and performing scheduled messages sequences from your Ruby on Rails application.</code></p>

<p>The one-liner is the single hardest sentence to write in the whole document and it deserves disproportionate effort. Rewrite it ten times. It does more work than any other sentence in the project. Treat it like ad copy because it is ad copy.</p>

<h2 id="the-why-should-i-care-paragraph">The “why should I care” paragraph</h2>

<p>Two to four sentences, immediately after the hero. Not features. The problem it solves and who it’s for.</p>

<p>Every engineer instinct is to list capabilities here. Capabilities are for later. Relevance is for now. If a reader doesn’t see themselves in this paragraph they close the tab and you never get them back. The tab is closed. It’s over. Answer: “This is the pain you’re facing and this is how I made it less painful.”</p>

<p>Caffeinate offers the typical tragedy that it solves.</p>

<h2 id="a-code-example-or-screenshot-fast">A code example or screenshot, fast</h2>

<p>Show, don’t tell.</p>

<p>For a library this is usually a minimal “here’s what using it looks like” snippet that fits on a laptop screen without scrolling. The example should be the most flattering possible representation of the library—the simplest case that still demonstrates the value prop. Not the most powerful case. Not the most complete case. The most flattering one. Stripe’s docs are the canonical reference here and it’s not an accident they’re cited every time someone writes a post like this.</p>

<p>If the project is a platform and not a library, use the most glamorous and story-telling screenshot possible.</p>

<h2 id="install-in-one-block-copy-pasteable">Install in one block, copy-pasteable</h2>

<p>If I were to rewrite Devise, I would show:</p>

<pre><code class="language-shell">bundle add devise 
rails g devise:install
rails g devise User
rails db:migrate
rails s
open "http://localhost:3000/users/sign_in"
</code></pre>

<p>Don’t make them think.</p>

<p>If installation genuinely requires more than one step, the marketer will lobby hard to make the first step the one that produces a visible result, and push the rest to a setup guide. “Do this and you’ll see something” is the goal. “Do these nine things and then you’ll be ready to do something” is how people bounce.</p>

<h2 id="a-quickstart-or-5-minute-tour">A quickstart or 5-minute tour</h2>

<p>Next is “oh I see why this is useful.”</p>

<p>This is where most READMEs go off the rails. The engineer instinct is to be comprehensive—show every option, every config, every edge case. The marketing instinct is to be ruthlessly selective and link out for everything else. The quickstart isn’t a feature tour. It’s a success moment. One. Just one. You’re trying to deliver dopamine, not completeness.</p>

<h2 id="features-framed-as-benefits">Features, framed as benefits</h2>

<p>If features appear at all, they’re written as “X so that Y,” not “supports X.” Bullets are fine. Three to seven of them. Each one sentence.</p>

<p>“Supports websockets” is engineer copy. “Real-time updates without polling, so your dashboards feel instant” is what the marketer rewrites it to. Same feature. One of those sentences makes someone want to try it. The other one makes someone scroll past.</p>

<h2 id="social-proof">Social proof</h2>

<p>Logos of companies using it. GitHub star count if it’s good. Testimonial quotes from known developers. Links to talks and posts about it.</p>

<p>Engineers undervalue this badly. Marketers know it does enormous work. Even a single “used in production at $COMPANY” line shifts perception, because the reader’s brain stops asking “is this a toy” and starts asking “how do I use this.” That’s a different question and you want them asking it.</p>

<h2 id="comparison-or-positioning">Comparison or positioning</h2>

<p>If there are obvious alternatives, address them. A short “how is this different from X” section.</p>

<p>Marketers love this because it answers the question already forming in the reader’s head. Engineers avoid it out of politeness. The compromise is a factual, generous-toned comparison rather than a hit piece. “X is great for Y; this is better for Z.” You’re not dunking, you’re orienting. The reader is going to make this comparison anyway—it’s better if you frame it than if they Google it.</p>

<h2 id="documentation-contribution-license-credits">Documentation, contribution, license, credits</h2>

<p>At the bottom, in that order. By this point the reader has either bounced or converted. This section is for the converted.</p>

<h2 id="the-cross-cutting-stuff-a-marketer-would-harp-on">The cross-cutting stuff a marketer would harp on</h2>

<p>A few things that aren’t sections but apply everywhere.</p>

<p>The README is not the docs. It is an ad for the docs. Every instinct to be thorough is the wrong instinct. Cut. Link out. Cut again. If something matters but doesn’t fit the funnel, it goes in a separate doc. The README is the trailer, not the movie.</p>

<p>Voice matters more than developers think. A README with a distinct voice—even a slightly opinionated or funny one—outperforms a neutral one. Tailwind, Bun, htmx, Astro. All have personality. It’s not a coincidence. Neutral copy reads like it was written by committee because it usually was, and people can tell.</p>

<p>Visuals carry disproportionate weight. A single well-chosen gif demonstrating the library in action is worth roughly an entire feature list. If the thing can be shown, show it. The gif is the most efficient unit of communication available to you and it costs 20 minutes of recording to make one.</p>

<p>Optimize for the skim. Assume the reader gives the README eight seconds before deciding whether to keep reading. Eight. Short paragraphs. Put the most important thing first in every section. If your first sentence in a section is “in order to understand X, it’s helpful to first consider Y,” congrats, they’re gone.</p>

<h2 id="what-to-actually-go-look-at">What to actually go look at</h2>

<p>If you want a worked example of any of this, the READMEs for Bun, Tailwind, Zod, htmx, and Drizzle ORM are all closer to landing page than docs and would be cited as benchmarks by anyone who thinks about this seriously. Stripe and Linear’s broader docs sites are the gold standard for the same instincts applied at scale.</p>

<p>The depressing part, if you’re a maintainer who’s been writing READMEs the engineer way for a decade, is that most of this isn’t hard. It’s just different. You don’t need to be a writer. You need to spend an afternoon thinking about your project the way a stranger sees it for the first time, and rewrite accordingly.</p>

<p>The README is the only part of your project most people will ever experience. It’s worth being weird about.</p>
]]></content>
    
    <summary>I have a few open-source libraries that you could call popular. I use a lot of others.
</summary>
    
    
    <category term="open-source"/>
    
    <category term="writing"/>
    
  </entry>
  
  <entry>
    <title>Wide tables and query validation</title>
    <link href="/posts/wide-tables-and-query-validation/" rel="alternate" type="text/html"/>
    <id>/posts/wide-tables-and-query-validation/</id>
    <published>2026-01-03T00:00:00+00:00</published>
    <updated>2026-01-03T00:00:00+00:00</updated>
    <content type="html"><![CDATA[<p>I’m building an ETL system that imports billions of person records from data vendors. Each vendor sends different attributes—household income, automotive interests, education level, charitable donor status—in their own formats with their own column names. The system normalizes everything into a canonical schema and exposes a query interface for building audiences.</p>

<p>The obvious choice for storing hundreds of optional attributes is JSONB or EAV. Flexible, no migrations needed, handles sparse data well. I went with wide tables instead: every queryable attribute is a column on the <code>people</code> table.</p>

<p>This sounds like a 2005 decision. But Postgres handles wide tables fine—you can have up to 1,600 columns, and everything feels normal until you hit about 500 columns. The real win is developer experience: you get actual columns, ActiveRecord just works, queries are plain SQL.</p>

<h2 id="why-not-jsonb-or-eav">Why not JSONB or EAV?</h2>

<p>You could absolutely build validation on top of JSONB or EAV. The schema registry approach works with any storage backend. But wide tables give you a better developer experience.</p>

<p>With JSONB, your queries look like this:</p>

<pre><code class="language-ruby">Person.where("attributes-&gt;&gt;'household_income' = ?", "150k_to_200k")
</code></pre>

<p>With wide tables:</p>

<pre><code class="language-ruby">Person.where(household_income: "150k_to_200k")
</code></pre>

<p>The second version works with ActiveRecord’s query interface, supports scopes, plays nice with Arel, and doesn’t require remembering JSON path syntax. Your editor can autocomplete column names. Your database can index them directly. <code>rails console</code> tab-completion works.</p>

<p>EAV has similar ergonomic problems—you’re always joining, always filtering on key/value pairs, always one abstraction away from the data.</p>

<p>Wide tables mean your attributes are just columns. Normal columns. The kind ActiveRecord was built for.</p>

<h2 id="the-query-interface">The query interface</h2>

<p>I built a query interface that validates before it queries:</p>

<pre><code class="language-ruby">People.where(household_income: "150k_to_200k")
      .where(individual_charitable_donor: true)
      .near(latitude: 40.7128, longitude: -74.0060, miles: 25)
      .count
</code></pre>

<p>This looks like ActiveRecord, but it’s not. It’s a query builder that checks every attribute and value against a canonical registry before generating SQL.</p>

<p>Try to query an attribute that doesn’t exist:</p>

<pre><code class="language-ruby">People.where(fake_attribute: "whatever")
# =&gt; People::UnknownAttributeError: unknown attribute: fake_attribute
</code></pre>

<p>Try to query an attribute that exists but isn’t enabled:</p>

<pre><code class="language-ruby">People.where(individual_language: "english")
# =&gt; People::AttributeNotActiveError: individual_language exists but is not active.
#    Add it to CanonicalAttributes::ACTIVE to enable.
</code></pre>

<p>Try to query with an invalid value:</p>

<pre><code class="language-ruby">People.where(household_income: "rich")
# =&gt; People::InvalidAttributeValueError: household_income must be one of:
#    under_20k, 20k_to_30k, 30k_to_40k, ... Got: "rich"
</code></pre>

<p>The query never runs. The error happens at the interface, with a message that tells you exactly what went wrong and what values are allowed.</p>

<h2 id="the-canonical-registry">The canonical registry</h2>

<p>The validation layer works because there’s a single source of truth for what attributes exist and what values they accept. It’s a Ruby hash:</p>

<pre><code class="language-ruby">module CanonicalAttributes
  ACTIVE = [
    :household_income,
    :individual_charitable_donor,
    :individual_automotive_purchases,
    # ... enabled attributes
  ]

  ALL = {
    household_income: {
      type: :text,
      description: "Estimated household income bracket",
      values: %w[
        under_20k 20k_to_30k 30k_to_40k 40k_to_50k
        50k_to_60k 60k_to_75k 75k_to_100k 100k_to_125k
        125k_to_150k 150k_to_200k 200k_to_250k 250k_to_500k
        500k_plus
      ],
      validates: [
        { bracket: { type: :income } }
      ]
    },

    individual_age: {
      type: :integer,
      description: "Estimated age of the individual (18-99)",
      validates: [
        { numericality: { greater_than_or_equal_to: 18, less_than_or_equal_to: 99 } }
      ]
    },

    individual_gender: {
      type: :text,
      values: %w[male female],
      validates: [
        { inclusion: { in: %w[male female] } }
      ]
    },

    # ... hundreds more
  }
end
</code></pre>

<p>Sure, not sexy. I thought about:</p>

<pre><code class="language-ruby">attribute :household_income, :text, description: "Estimated household income bracket", validations: { bracket: { type: :income } }, values: %w[ ... ]
</code></pre>

<p>And I might go there, idk. But this feels fine for now, and there’s less runtime overhead—which is negligible until you scale.</p>

<p>This registry does three things.</p>
<ol>
  <li>It defines what exists—if it’s not in <code>ALL</code>, it’s not a valid attribute.</li>
  <li>It controls what’s queryable—if it’s not in <code>ACTIVE</code>, you can’t query it yet, maybe the column hasn’t been added, maybe the data isn’t populated.</li>
  <li>It specifies valid values—the <code>validates</code> key mirrors ActiveModel validations, but runs against query values instead of record values.</li>
</ol>

<p>The same registry drives migrations. When I add a new attribute, a generator reads the registry and creates the migration:</p>

<pre><code class="language-ruby">add_column :people, :household_income, :string
add_index :people, :household_income, where: "household_income IS NOT NULL"
</code></pre>

<p>Schema, validation rules, and column definitions all come from the same place. They can’t diverge.</p>

<h2 id="validation-at-query-time">Validation at query time</h2>

<p>The query builder runs validation before it builds SQL:</p>

<pre><code class="language-ruby">def add_filter(key, value, type)
  if @canonical_attributes.key?(key.to_s)
    validate_attribute_value!(key, value)
    @attribute_filters &lt;&lt; { key: key, value: value, type: type }
  elsif CanonicalAttributes::REGISTRY.key?(key.to_sym)
    raise People::AttributeNotActiveError,
          "#{key} exists but is not active."
  else
    raise People::UnknownAttributeError, "unknown attribute: #{key}"
  end
end
</code></pre>

<p>The validation method handles different value types—arrays, ranges, hashes for comparisons:</p>

<pre><code class="language-ruby">def validate_attribute_value!(key, value)
  config = CanonicalAttributes::REGISTRY[key.to_sym]
  return unless config

  validates = config[:validates]
  return unless validates

  values_to_check = case value
                    when Array then value
                    when Range then [value.begin, value.end].compact
                    when Hash then value.values
                    else [value]
                    end

  values_to_check.each do |val|
    validate_single_value!(key, val, config, validates)
  end
end
</code></pre>

<p>This means all these query styles get validated:</p>

<pre><code class="language-ruby"># Exact match
People.where(household_income: "100k_to_125k")

# Multiple values (OR)
People.where(household_income: ["100k_to_125k", "125k_to_150k"])

# Comparisons
People.where(individual_age: { gte: 25, lte: 45 })

# Ranges
People.where(individual_age: 25..45)

# Negation
People.where.not(household_income: "under_20k")
</code></pre>

<p>Every value in every query form gets checked. If you pass <code>individual_age: { gte: 150 }</code>, you get an error before the query runs.</p>

<h2 id="the-result-interface">The result interface</h2>

<p>The queries return wrapped results, not ActiveRecord objects:</p>

<pre><code class="language-ruby">person = People.where(household_income: "150k_to_200k").first

person.household_income     # =&gt; "150k_to_200k"
person.first_name           # =&gt; "Jane"
person.city                 # =&gt; "houston"
person.emails               # =&gt; [{ "email" =&gt; "jane@example.com", ... }]
</code></pre>

<p>It looks like a model, but it’s a read-only projection. You can’t <code>person.update!</code>—this is a query interface, not an ORM. The underlying <code>Person</code> model exists, but you get to it explicitly with <code>person.to_model</code> if you need it.</p>

<p>(In case you were wondering: <code>ActiveRecord::Persistence</code> isn’t used for any of these attributes; there’s a separate pipeline for them entirely using some deep Postgres stuff.)</p>

<p>The class inspection shows you what’s available:</p>

<pre><code class="language-ruby">People::Result.inspect
# =&gt; "People::Result(id: integer, first_name: string, last_name: string,
#     city: string, state: string, household_income: text, ...)"
</code></pre>

<h2 id="try-it">Try it</h2>

<p>The examples dropdown shows valid queries, invalid values, unknown attributes, and inactive attributes—each fails differently, before any SQL runs.</p>

<script type="text/javascript">
  window.CDN_HOST = 'https://cdn.josh.mn'
</script>

<div class="playground" data-runtime="rails">
<script type="application/json" class="playground-config">
{"global":{"schema":"create_table :people, force: true do |t|\n  t.string :first_name\n  t.string :last_name\n  t.string :city\n  t.string :state\n  t.decimal :latitude, precision: 10, scale: 6\n  t.decimal :longitude, precision: 10, scale: 6\n  t.string :household_income\n  t.integer :individual_age\n  t.string :individual_gender\n  t.boolean :individual_charitable_donor\n  t.string :individual_automotive_purchases\n  t.string :individual_education_level\n  t.string :individual_language\n  t.timestamps\nend\n\nadd_index :people, :household_income, where: \"household_income IS NOT NULL\"\nadd_index :people, :individual_age, where: \"individual_age IS NOT NULL\"\nadd_index :people, :individual_gender, where: \"individual_gender IS NOT NULL\"\nadd_index :people, :individual_charitable_donor, where: \"individual_charitable_donor IS NOT NULL\"\n","models":"module CanonicalAttributes\n  ACTIVE = [\n    :household_income,\n    :individual_age,\n    :individual_gender,\n    :individual_charitable_donor,\n    :individual_automotive_purchases,\n    :individual_education_level,\n  ]\n\n  ALL = {\n    household_income: {\n      type: :text,\n      description: \"Estimated household income bracket\",\n      values: %w[\n        under_20k 20k_to_30k 30k_to_40k 40k_to_50k\n        50k_to_60k 60k_to_75k 75k_to_100k 100k_to_125k\n        125k_to_150k 150k_to_200k 200k_to_250k 250k_to_500k\n        500k_plus\n      ]\n    },\n    individual_age: {\n      type: :integer,\n      description: \"Estimated age of the individual (18-99)\",\n      range: 18..99\n    },\n    individual_gender: {\n      type: :text,\n      description: \"Gender of the individual\",\n      values: %w[male female]\n    },\n    individual_charitable_donor: {\n      type: :boolean,\n      description: \"Whether the individual is a charitable donor\"\n    },\n    individual_automotive_purchases: {\n      type: :text,\n      description: \"Recent automotive purchase type\",\n      values: %w[new_car used_car luxury_car truck suv none]\n    },\n    individual_education_level: {\n      type: :text,\n      description: \"Highest education level\",\n      values: %w[high_school some_college bachelors masters doctorate]\n    },\n    individual_language: {\n      type: :text,\n      description: \"Primary language (not currently active)\",\n      values: %w[english spanish french german mandarin]\n    }\n  }\nend\n\nclass Person < ApplicationRecord\nend\n\nmodule People\n  class UnknownAttributeError < StandardError; end\n  class AttributeNotActiveError < StandardError; end\n  class InvalidAttributeValueError < StandardError; end\n\n  class ResultSet < Array\n    def inspect\n      if empty?\n        \"[]\"\n      else\n        \"[\\n\" + map { |r| \"  #{r.inspect}\" }.join(\",\\n\") + \"\\n]\"\n      end\n    end\n  end\n\n  class Result\n    attr_reader :attributes\n\n    def initialize(record)\n      @attributes = record.attributes\n    end\n\n    def method_missing(name, *args)\n      if @attributes.key?(name.to_s)\n        @attributes[name.to_s]\n      else\n        super\n      end\n    end\n\n    def respond_to_missing?(name, include_private = false)\n      @attributes.key?(name.to_s) || super\n    end\n\n    def inspect\n      attrs = @attributes.except(\"created_at\", \"updated_at\")\n                         .compact\n                         .map { |k, v| \"#{k}: #{v.inspect}\" }\n                         .join(\", \")\n      \"#<People::Result #{attrs}>\"\n    end\n\n    alias_method :to_s, :inspect\n  end\n\n  class Query\n    def initialize\n      @scope = Person.all\n      @canonical_attributes = CanonicalAttributes::ACTIVE.map(&:to_s)\n    end\n\n    def where(conditions)\n      conditions.each do |key, value|\n        add_filter(key, value)\n      end\n      self\n    end\n\n    def not(conditions)\n      conditions.each do |key, value|\n        add_filter(key, value, :not)\n      end\n      self\n    end\n\n    def near(latitude:, longitude:, miles:)\n      @scope = @scope.where(\"1=1\")\n      self\n    end\n\n    def count\n      @scope.count\n    end\n\n    def first\n      record = @scope.first\n      record ? Result.new(record) : nil\n    end\n\n    def all\n      ResultSet.new(@scope.map { |r| Result.new(r) })\n    end\n\n    def to_a\n      all\n    end\n\n    private\n\n    def add_filter(key, value, type = :eq)\n      key_s = key.to_s\n      key_sym = key.to_sym\n\n      if @canonical_attributes.include?(key_s)\n        validate_attribute_value!(key_sym, value)\n        if type == :not\n          @scope = @scope.where.not(key => value)\n        else\n          @scope = @scope.where(key => value)\n        end\n      elsif CanonicalAttributes::ALL.key?(key_sym)\n        raise AttributeNotActiveError,\n              \"#{key} exists but is not active. Add it to CanonicalAttributes::ACTIVE to enable.\"\n      else\n        raise UnknownAttributeError, \"unknown attribute: #{key}\"\n      end\n    end\n\n    def validate_attribute_value!(key, value)\n      config = CanonicalAttributes::ALL[key]\n      return unless config\n\n      values_to_check = case value\n                        when Array then value\n                        when Range then [value.begin, value.end].compact\n                        when Hash then value.values\n                        else [value]\n                        end\n\n      values_to_check.each do |val|\n        validate_single_value!(key, val, config)\n      end\n    end\n\n    def validate_single_value!(key, val, config)\n      case config[:type]\n      when :text\n        if config[:values] && !config[:values].include?(val.to_s)\n          raise InvalidAttributeValueError,\n                \"#{key} must be one of: #{config[:values].join(', ')}. Got: #{val.inspect}\"\n        end\n      when :integer\n        unless val.is_a?(Integer)\n          raise InvalidAttributeValueError,\n                \"#{key} must be an integer. Got: #{val.inspect}\"\n        end\n        if config[:range] && !config[:range].cover?(val)\n          raise InvalidAttributeValueError,\n                \"#{key} must be between #{config[:range].begin} and #{config[:range].end}. Got: #{val}\"\n        end\n      when :boolean\n        unless [true, false].include?(val)\n          raise InvalidAttributeValueError,\n                \"#{key} must be true or false. Got: #{val.inspect}\"\n        end\n      end\n    end\n  end\n\n  class << self\n    def where(conditions)\n      Query.new.where(conditions)\n    end\n\n    def count\n      Query.new.count\n    end\n\n    def first\n      Query.new.first\n    end\n\n    def all\n      Query.new.all\n    end\n  end\nend\n","seed":"ActiveRecord::Base.logger = Logger.new(STDOUT)\nActiveRecord::Base.logger.formatter = proc { |_, _, _, msg| \"#{msg.strip}\\n\" }\n\nPerson.create!(\n  first_name: \"Jane\",\n  last_name: \"Smith\",\n  city: \"houston\",\n  state: \"TX\",\n  latitude: 29.7604,\n  longitude: -95.3698,\n  household_income: \"150k_to_200k\",\n  individual_age: 34,\n  individual_gender: \"female\",\n  individual_charitable_donor: true,\n  individual_automotive_purchases: \"luxury_car\",\n  individual_education_level: \"masters\"\n)\n\nPerson.create!(\n  first_name: \"John\",\n  last_name: \"Doe\",\n  city: \"new york\",\n  state: \"NY\",\n  latitude: 40.7128,\n  longitude: -74.0060,\n  household_income: \"100k_to_125k\",\n  individual_age: 42,\n  individual_gender: \"male\",\n  individual_charitable_donor: false,\n  individual_automotive_purchases: \"suv\",\n  individual_education_level: \"bachelors\"\n)\n\nPerson.create!(\n  first_name: \"Maria\",\n  last_name: \"Garcia\",\n  city: \"los angeles\",\n  state: \"CA\",\n  latitude: 34.0522,\n  longitude: -118.2437,\n  household_income: \"75k_to_100k\",\n  individual_age: 28,\n  individual_gender: \"female\",\n  individual_charitable_donor: true,\n  individual_automotive_purchases: \"used_car\",\n  individual_education_level: \"bachelors\"\n)\n\nPerson.create!(\n  first_name: \"Robert\",\n  last_name: \"Johnson\",\n  city: \"chicago\",\n  state: \"IL\",\n  latitude: 41.8781,\n  longitude: -87.6298,\n  household_income: \"200k_to_250k\",\n  individual_age: 55,\n  individual_gender: \"male\",\n  individual_charitable_donor: true,\n  individual_automotive_purchases: \"luxury_car\",\n  individual_education_level: \"doctorate\"\n)\n\nPerson.create!(\n  first_name: \"Emily\",\n  last_name: \"Chen\",\n  city: \"san francisco\",\n  state: \"CA\",\n  latitude: 37.7749,\n  longitude: -122.4194,\n  household_income: \"250k_to_500k\",\n  individual_age: 31,\n  individual_gender: \"female\",\n  individual_charitable_donor: false,\n  individual_automotive_purchases: \"new_car\",\n  individual_education_level: \"masters\"\n)\n","examples":[{"label":"Query by income bracket","code":"puts People.where(household_income: \"150k_to_200k\").first\n"},{"label":"Query charitable donors","code":"puts People.where(individual_charitable_donor: true).count\n"},{"label":"Invalid value error","code":"puts People.where(household_income: \"rich\")\n"},{"label":"Unknown attribute error","code":"puts People.where(fake_attribute: \"whatever\")\n"},{"label":"Inactive attribute error","code":"puts People.where(individual_language: \"english\")\n"},{"label":"Invalid age range","code":"puts People.where(individual_age: 150)\n"},{"label":"Multiple values (OR)","code":"puts People.where(household_income: [\"100k_to_125k\", \"125k_to_150k\"]).all\n"},{"label":"Chained conditions","code":"puts People.where(individual_charitable_donor: true)\n      .where(individual_gender: \"female\")\n      .all\n"},{"label":"All people","code":"puts People.all\n"},{"label":"View canonical registry","code":"puts CanonicalAttributes::ALL.keys\n"}]},"instance":{"setup":"@dog = \"hello\"\n"},"examples":[{"label":"Query by income bracket","code":"puts People.where(household_income: \"150k_to_200k\").first\n"},{"label":"Query charitable donors","code":"puts People.where(individual_charitable_donor: true).count\n"},{"label":"Invalid value error","code":"puts People.where(household_income: \"rich\")\n"},{"label":"Unknown attribute error","code":"puts People.where(fake_attribute: \"whatever\")\n"},{"label":"Inactive attribute error","code":"puts People.where(individual_language: \"english\")\n"},{"label":"Invalid age range","code":"puts People.where(individual_age: 150)\n"},{"label":"Multiple values (OR)","code":"puts People.where(household_income: [\"100k_to_125k\", \"125k_to_150k\"]).all\n"},{"label":"Chained conditions","code":"puts People.where(individual_charitable_donor: true)\n      .where(individual_gender: \"female\")\n      .all\n"},{"label":"All people","code":"puts People.all\n"},{"label":"View canonical registry","code":"puts CanonicalAttributes::ALL.keys\n"}],"autorun":"puts People.where(household_income: \"150k_to_200k\").first"}
</script>
<div class="playground-toolbar"><select class="playground-examples"><option value="">-- examples --</option></select></div>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">puts</span> <span class="no">People</span><span class="p">.</span><span class="nf">where</span><span class="p">(</span><span class="ss">household_income: </span><span class="s2">"150k_to_200k"</span><span class="p">).</span><span class="nf">first</span></code></pre></div></div>
<div class="playground-warning d-none">This does not play nicely with iOS.</div>
<div class="playground-controls">
<div class="playground-controls-buttons">
<button class="playground-run" disabled="">Run</button>
<button class="playground-reset" disabled="">Reset</button>
</div>
<span class="playground-controls-note">loads ~76MB Rails environment</span>
</div>
<div class="playground-output"><pre><code><span style="font-weight:700"><span style="color:#4ec9b0">Person Load (0.3ms)</span>  <span style="font-weight:700"><span style="color:#569cd6">SELECT "people".* FROM "people" WHERE "people"."household_income" = '150k_to_200k' ORDER BY "people"."id" ASC LIMIT 1</span>
#&lt;People::Result id: 1, first_name: "Jane", last_name: "Smith", city: "houston", state: "TX", latitude: 0.297604e2, longitude: -0.953698e2, household_income: "150k_to_200k", individual_age: 34, individual_gender: "female", individual_charitable_donor: true, individual_automotive_purchases: "luxury_car", individual_education_level: "masters"&gt;
=&gt; nil</span></span></code></pre></div>
</div>

<h2 id="trade-offs">Trade-offs</h2>

<p>Wide tables have real costs.</p>

<p>Every new queryable attribute requires a migration. In a system with hundreds of attributes across multiple vendors, this means more deploys. That’s fine, deploys are cheap. I’ve mitigated this with a generator that reads the registry and produces the migration, but it’s still a deploy.</p>

<p>Sparse data means lots of NULLs. Most people don’t have values for most attributes. Postgres handles this well—NULL storage is cheap, about 1 bit per column in the null bitmap—but it’s not as compact as JSONB for very sparse data.</p>

<p>Schema changes require coordination. If two vendors need new attributes at the same time, the migrations need to be sequenced. With JSONB, you’d just start writing different keys.</p>

<p>These are real trade-offs. I chose wide tables anyway because the developer experience wins outweigh the operational costs for my use case.</p>

<h2 id="why-this-works">Why this works</h2>

<p>The registry is the key. It’s not just documentation—it’s the schema, the validation rules, and the query contract in one place. The same hash that defines valid query values also drives migrations, generates API documentation, and powers the import normalizers.</p>

<p>When you add an attribute, you add it to the registry. A generator creates the migration. The query interface automatically validates it. The import pipeline knows how to normalize vendor data into canonical values. Nothing can diverge because everything reads from the same source.</p>

<p>Wide tables make the schema explicit. Your columns are your contract. ActiveRecord understands them. Your database can optimize for them. Your tests can assert on them. There’s no layer of indirection between “what attributes exist” and “what columns are in the database.”</p>

<p>The flexibility of JSONB is real. But so is the friction of working one abstraction away from your data. Wide tables put your attributes where Rails expects them to be.</p>

<p>JSONB and EAV are the right tools for plenty of problems. I considered both and decided they weren’t right for this one. I could be wrong—I often am—but a month in, the developer experience has been worth the migration overhead.</p>
]]></content>
    
    <summary>I’m building an ETL system that imports billions of person records from data vendors. Each vendor sends different attributes—household income, automotive interests, education level, charitable donor status—in their own formats with their own column names. The system normalizes everything into a canonical schema and exposes a query interface for building audiences.
</summary>
    
    
    <category term="ruby"/>
    
  </entry>
  
  <entry>
    <title>I got out of federal prison and couldn&apos;t log into my GitHub</title>
    <link href="/posts/i-got-out-of-federal-prison-and-couldnt-log-into-my-github/" rel="alternate" type="text/html"/>
    <id>/posts/i-got-out-of-federal-prison-and-couldnt-log-into-my-github/</id>
    <published>2025-12-28T00:00:00+00:00</published>
    <updated>2025-12-28T00:00:00+00:00</updated>
    <content type="html"><![CDATA[<p>I got out of prison and couldn’t log into my GitHub.</p>

<p>The password while I was gone. The phone number tied to the account got transferred to another carrier. The device with my two-factor authentication codes—gone. Eighteen months away, and I came back to find myself locked out of my own account.</p>

<p>GitHub, for those unfamiliar, is where programmers store and share code. It’s the largest platform of its kind. Your GitHub account is your professional identity if you work in software. It’s your portfolio, your history, your proof of work. Mine had twelve years of contributions on it. Every project I’d built, every line I’d written, every collaboration I’d been part of.</p>

<p>The obvious response is “just make a new one.” It’s not that simple. That account has two nearly-finished books in private repositories—projects I’d been working on for years. Intellectual property I was developing. Contractual work I’m still legally responsible for delivering. Libraries that other developers depend on, and those have gone stale in my absence because I can’t push updates. The username itself—joshmn—is mine in a way that matters to me, even if it shouldn’t, but that’s beside the point. It’s how people in my industry know me. Starting over with joshmn2 or realJoshBrody isn’t really starting over. It’s admitting that the confirm-your-identity process is correct and practical and reasonable.</p>

<p>When I got out of federal prison, these affected most of my digital life—this domain (the only one that really matters to me), social media accounts (those are throwawayable), and some other less notable things. I was able to recover those after a lengthy and sometimes painful process. But others have proven difficult, if not unreasonable—to a degree.</p>

<p>Update: On January 13, 2026 I was restored access to my GitHub. My feelings about 2FA persist.</p>

<hr />

<h2 id="every-method-failed">Every method failed</h2>

<p>GitHub offers recovery methods. They’re reasonable. They all failed.</p>

<p>API key verification? I had an old personal access token associated with my account. I provided it. GitHub said it was expired and therefore didn’t count. The logic is sound—an expired key could have been compromised, leaked, harvested from some old breach. They can’t trust it. But I had it. I knew it. It was mine.</p>

<p>SMS verification? The phone number tied to my account is gone. Transferred to another provider after it went stale. I can’t receive a text at a number that’s no longer mine.</p>

<p>Two-factor authentication codes? The device that had my authenticator app was gone. The backup codes were printed on paper that I have to assume was thrown out or destroyed. Maybe they’re sitting in a landfill somewhere unshredded. Maybe they were burned. Either way, I don’t have them.</p>

<p>Each method makes sense in isolation. Each method assumes you still have access to something you set up years ago—a phone number, a device, a piece of paper. The system is designed for people who lose one factor at a time, not all of them at once. I lost everything simultaneously. Not because I was careless, but because I went to prison and someone I trusted decided to take what was mine.</p>

<hr />

<h2 id="this-could-be-you">This could be you</h2>

<p>I know what you’re thinking. This is a prison story. This doesn’t apply to me. But you don’t have to go to prison to end up here.</p>

<p>Your fault-ish: Your phone gets stolen on vacation. You’re in Barcelona, you set your bag down for thirty seconds, and it’s gone. Your whole digital life was in that bag. Your authenticator app, your SIM card, your backup codes saved in a notes app you thought was clever.</p>

<p>Less your fault: house fire. Everything you own is ash. Including the drawer where you kept that piece of paper with your recovery codes, the one you printed out three years ago and never thought about again.</p>

<p>Maybe your fault: nasty divorce. Your ex won’t hand over the iPad—the one that has your authenticator app on it, the one you set up back when you shared everything and didn’t think about what would happen when you didn’t.</p>

<p>Your brain’s fault: you switch phones. You’re excited about the new one. You restore from backup, everything seems fine, and then you wipe the old phone before you realize your authenticator app doesn’t transfer automatically. The codes are gone. You didn’t even know they were stored locally.</p>

<p>Not your fault, kind of: you get laid off. You return the work laptop and the work phone—the one that had your personal authenticator on it, because who keeps work and personal separate anymore? IT wipes the device before you remember what was on it.</p>

<p>Sad, hopefully not your fault: your parent dies, and they were your emergency contact, your backup plan, the person who had the envelope with your codes in their safe. Now their estate is in probate and you can’t access anything.</p>

<p>My point is that almost nobody writes down backup codes. Everyone assumes they’ll have access to their phone forever. Everyone assumes their circumstances won’t change dramatically. Until they do.</p>

<p>You are not your phone.</p>

<hr />

<h2 id="i-asked-the-internet">I asked the internet</h2>

<p>I posted on <a href="https://news.ycombinator.com/item?id=45451567">Hacker News</a> asking for help. Hacker News is a forum popular with programmers and startup founders—the kind of people who think carefully about security, who have opinions about encryption, and the type of folk who ultimately placed their opinions on how we identify ourselves by building these systems originally after brainstorming them over coffee. They’re mostly a skeptical-by-default crowd. If you want to know whether your take is defensible, post it there and wait for someone to tell you why you’re wrong. While it may sound like reddit in its existence and description, it could not be more unlike reddit in it’s community.</p>

<p>The post got over a hundred comments (this is a lot on Hacker News). People debated both sides.</p>

<p>The top response cut right to it: “The situation you are in is very unfortunate and I am sympathetic but in GitHub’s defence, this is exactly what I hope would happen when I enable 2FA. I would be very perturbed to find out that GitHub would grant access to my account given identity documents.”</p>

<p>I sat with that for a while. Then I replied: “That’s the same stance I have and why I’m torn.”</p>

<p>I meant it. I still do.</p>

<hr />

<h2 id="everyone-had-ideas">Everyone had ideas</h2>

<p>The comment section was full of suggestions. Smart people trying to solve an unsolvable problem.</p>

<p>Put your ID on file ahead of time—driver’s license, passport, something official. That way, if you ever lose access, you can verify with government documents. Sounds reasonable. But think about what that actually means. You’ve now turned two-factor authentication into single-factor authentication. The ID becomes the master key that bypasses everything else. Anyone who can forge or steal your ID—or coerce you into handing it over—now has a backdoor into your account. That’s not more secure. That’s less secure with extra steps.</p>

<p>Offer in-person verification for a fee. Fly to GitHub’s headquarters in San Francisco, show up with your passport, sit across from a human being, pay whatever it costs. Sounds reasonable. Doesn’t scale. GitHub has over 100 million users. They’re owned by Microsoft now and even they can’t run an identity verification desk for everyone who loses their second factor. The math doesn’t work. And who’s qualified to verify identity anyway? What training would that person have? What liability would they take on?</p>

<p>Get a court order. Have a judge confirm you are who you say you are, legally and officially, with the full weight of the judicial system behind it. Sounds reasonable. Also: it’s a GitHub account, and that doesn’t mean it’s still <em>my</em> GitHub account. A place where I store code. The fact that “get a federal judge involved” is even on the table tells you something about how deeply broken this situation is. How did we get here? How did we build a system where recovering your own property requires the intervention of the courts?</p>

<p>And yet—here I am. That’s exactly what I’m doing.</p>

<hr />

<h2 id="lawyers-for-a-github-account">Lawyers, for a GitHub account</h2>

<p>I have lawyers involved now. Real lawyers, from a real law firm, billing real hours. They contacted GitHub’s legal team. The first suggestion from GitHub was that I could ask my probation officer to request that a judge issue an order confirming my identity. For a GitHub account. The lawyers consulted with criminal defense attorneys in the relevant jurisdiction and determined that approach was unlikely to work—judges don’t typically issue orders like that at a probation officer’s request.</p>

<p>So we pivoted. The solution currently on the table: a notarized affidavit, sworn under penalty of perjury, attesting that I am who I say I am and that the account belongs to me. Multiple government IDs—passport, state ID, the federal release identification card I was issued upon leaving prison. A list of private repository names that only the real account owner would know. An expired API token that was tied to the account. Oh, and a hang-myself clause: me acknowledging that if I’m lying, GitHub can refer the matter to law enforcement and my probation officer.</p>

<p>All of this. For a GitHub account.</p>

<hr />

<h2 id="this-is-by-design">This is by design</h2>

<p>Here’s the thing: this is correct behavior.</p>

<p>If someone was trying to social engineer their way into your account—if an attacker wanted access to your code, your private repositories, your credentials—you’d want GitHub to make it this hard. You’d want them to require a notarized affidavit. You’d want them to demand information only the real owner could know. You’d want them to ask for multiple forms of government identification and still be skeptical.</p>

<p>You’d want them to ignore the sob story.</p>

<p>Because attackers have sob stories too. They’re good at this. Social engineering is a craft. The email that says “I’m locked out of my account, my grandmother just died and I need access to her repository” might be true. Or it might be the opening move in a supply chain attack that compromises thousands of downstream systems. GitHub can’t tell the difference. They’re not supposed to be able to tell the difference. That’s the whole point.</p>

<p>The system can’t distinguish between “legitimate owner in an unusual situation” and “attacker with a compelling narrative.” If there were an escape hatch for sympathetic cases—some process where, if your story was sad enough, they’d let you in—that escape hatch is exactly what every attacker would exploit. The security model depends on there being no exceptions. My situation is unusual, but it’s not special. The rules apply to me because they have to apply to everyone.</p>

<hr />

<h2 id="github-accounts-are-infrastructure">GitHub accounts are infrastructure</h2>

<p>This isn’t just about me being locked out of my stuff. GitHub accounts are critical infrastructure for not only my career but also for the infrastructure of humanity—save for uncontacted tribes.</p>

<p>Think about how modern software gets built. Nobody writes everything from scratch anymore. You pull in open-source code that handles common tasks. That code is very likely stored on GitHub, most of it maintained by individual developers, often volunteers, often working alone. When you add this code to your code, you’re trusting that the person who maintains it is who they say they are, and the code is what it says it is.</p>

<p>One compromised maintainer account can push malicious code to a package that gets pulled into thousands of pieces of software downstream. A single bad update can propagate through the entire ecosystem in hours. We’ve seen this happen. <a href="https://www.cisa.gov/news-events/alerts/2021/10/22/malware-discovered-popular-npm-package-ua-parser-js">In 2021, a popular JavaScript package called <code>ua-parser-js</code> was compromised</a>; malicious code was pushed to millions of downstream users. <a href="https://cloud.google.com/blog/topics/threat-intelligence/supply-chain-node-js">The same year, the <code>coa</code> and <code>rc</code> packages were hijacked</a>. In 2024, <a href="https://www.wired.com/story/jia-tan-xz-backdoor/">a backdoor was discovered in xz Utils</a>, a compression library used in virtually every Linux distribution—the result of a years-long social engineering campaign to gain a maintainer’s trust.</p>

<p>These attacks are getting more sophisticated. The stakes are enormous. A major supply chain compromise could affect banking systems, hospitals, power grids—anything that runs on software, which is everything.</p>

<p>The paranoia isn’t paranoia. It’s proportional to the risk.</p>

<p>GitHub can’t afford to get this wrong. And “wrong” means giving access to someone who shouldn’t have it. From their perspective, being too strict is a feature, not a bug. The cost of locking out a legitimate user is annoying for that user. The cost of letting in an attacker is potentially catastrophic for everyone who depends on the software that attacker can now compromise.</p>

<p>I’m annoyed. But I understand.</p>

<hr />

<h2 id="the-thing-is-not-you">The thing is not you</h2>

<p>The deeper problem is philosophical, and it’s one we haven’t fully grappled with as a society.</p>

<p>Two-factor authentication doesn’t prove you are you. It proves you have access to a thing.</p>

<p>The thing is not you. It’s a phone, an app, a hardware key, a piece of paper with codes printed on it, a SIM card, a device. We call it “something you have,” as distinct from “something you know” (your password) and “something you are” (your fingerprint, your face). But “something you have” is precarious in a way we don’t like to admit. You can lose it. It can be stolen. It can break. It can be taken from you by someone you trusted. It can be destroyed in a fire or a flood. It can be confiscated by the state.</p>

<p>When the thing is gone—stolen, broken, wiped, transferred, burned, seized—you become indistinguishable from an attacker. You’re just someone claiming to be the account owner without the credentials to prove it. You can have a passport, a birth certificate, a social security card, a federal release ID, witnesses who will swear you are who you say you are. None of that matters. The system doesn’t verify identity. It verifies possession.</p>

<p>There’s no “I am obviously the real owner” escape hatch. There can’t be. That escape hatch would be a security vulnerability. The whole model depends on the authentication factors being the only path in. A forged passport and a smiling face means you can fly under the guise you are your passport.</p>

<p>We’ve built a world where access to a device is more authoritative than legal identity documents. Where losing a phone is more consequential than losing a passport. Where the thing in your pocket is more “you” than you are.</p>

<p>I’m not sure that’s wrong. I’m not sure it’s right either. But it’s where we are.</p>

<hr />

<h2 id="waiting">Waiting</h2>

<p>I’m still waiting to hear back on the affidavit. Maybe it works. Maybe GitHub’s legal team reviews everything, confirms the information matches what’s on their end, and restores my access. Maybe I get my account back, my twelve years of work, my books, my packages, my username.</p>

<p>Or maybe they find some reason to say no. Maybe there’s another hoop, another document, another verification step. Maybe this drags on for months. I don’t know. That’s fine, I’ll get it back eventually. But I’m at the mercy of a process I can’t control, trying to prove I’m me to a system designed to be skeptical of exactly that claim.</p>

<p>But the lesson here isn’t “GitHub’s recovery process is broken.” It’s not. The process is working exactly as designed. The lesson is that we’ve all made a trade-off we didn’t fully think through.</p>

<p>We added two-factor authentication because security matters. Because passwords get leaked, phished, guessed, reused across sites. Because we needed something stronger. And two-factor is stronger. It’s dramatically harder for an attacker to compromise an account protected by 2FA. That’s not nothing. That’s important.</p>

<p>But we didn’t think about the failure mode. We didn’t consider what happens when the second factor disappears. We didn’t ask ourselves: what’s the recovery path when everything goes wrong at once?</p>

<p>The answer, it turns out, is lawyers and affidavits and government IDs and still maybe not getting your account back. The answer is that there’s no good answer.</p>

<hr />

<h2 id="write-down-your-backup-codes">Write down your backup codes</h2>

<p>Write down your backup codes.</p>

<p>I’m serious. Do it today. Do it right now, before you forget, before you convince yourself you’ll get to it later.</p>

<p>Put them somewhere that isn’t your phone. Isn’t your computer. Isn’t in the cloud, where they could be compromised by the same breach that takes your account. Isn’t in the possession of someone who might betray you. Make copies of it, and bury it where you’d bury gold, and put it in a safe deposit box, and somewhere in the woods. A fireproof safe bolted to your floor. A sealed envelope with a lawyer. Best idea is tattooed on your inner thigh, but that’s compromisable too, I guess.</p>

<p>Just somewhere that will survive whatever your personal disaster turns out to be. Because if you lose them, you’re me. And being me is a lot of paperwork.</p>
]]></content>
    
    <summary>I got out of prison and couldn’t log into my GitHub.
</summary>
    
    
    <category term="personal"/>
    
  </entry>
  
  <entry>
    <title>Interactive Rails playgrounds</title>
    <link href="/posts/interactive-rails-playgrounds/" rel="alternate" type="text/html"/>
    <id>/posts/interactive-rails-playgrounds/</id>
    <published>2025-12-01T00:00:00+00:00</published>
    <updated>2025-12-01T00:00:00+00:00</updated>
    <content type="html"><![CDATA[<p>I built a WASM-compatible Rails playground that runs entirely in your browser. No server, no setup, just Rails. <a href="https://github.com/palkan/wasmify-rails">This is built on the backs of people much smarter than me</a>.</p>

<p>Try it out below. The VM takes a few seconds to load, but once it’s ready you can run any Active Record code against a real SQLite database.</p>

<script type="text/javascript">
  window.CDN_HOST = 'https://cdn.josh.mn'
</script>

<div class="playground" data-runtime="rails">
<script type="application/json" class="playground-config">
{"global":{"schema":"create_table :posts, force: true do |t|\n  t.string :title\n  t.text :body\n  t.boolean :published, default: false\n  t.timestamps\nend\n\ncreate_table :comments, force: true do |t|\n  t.references :post\n  t.string :author\n  t.text :content\n  t.timestamps\nend\n","models":"class Post < ApplicationRecord\n  has_many :comments, dependent: :destroy\n  scope :published, -> { where(published: true) }\n  scope :drafts, -> { where(published: false) }\n\n  def word_count\n    body.to_s.split.size\n  end\nend\n\nclass Comment < ApplicationRecord\n  belongs_to :post\nend\n","seed":"ActiveRecord::Base.logger = Logger.new(STDOUT)\nActiveRecord::Base.logger.formatter = proc { |_, _, _, msg| \"#{msg.strip}\\n\" }\n\npost1 = Post.create(title: 'Getting Started with Ruby', body: 'Ruby is a dynamic, open source programming language with a focus on simplicity and productivity.', published: true)\npost1.comments.create(author: 'Alice', content: 'Great intro!')\npost1.comments.create(author: 'Bob', content: 'Very helpful, thanks!')\n\npost2 = Post.create(title: 'Rails is Magic', body: 'Rails is a web application framework that includes everything needed to create database-backed web applications.', published: true)\npost2.comments.create(author: 'Charlie', content: 'Rails changed my life')\n\nPost.create(title: 'Draft Post', body: 'This is still a work in progress...', published: false)\n","examples":[{"label":"List all posts","code":"Post.all"},{"label":"Published posts only","code":"Post.published"},{"label":"Draft posts","code":"Post.drafts"},{"label":"First post with comments","code":"post = Post.first\nputs \"Post: #{post.title}\"\nputs \"Comments: #{post.comments.count}\"\npost.comments\n"},{"label":"Word count","code":"Post.all.map { |p| [p.title, p.word_count] }"},{"label":"Create a comment","code":"Post.first.comments.create(author: 'You', content: 'Hello from the REPL!')"}]},"instance":{},"examples":[{"label":"List all posts","code":"Post.all"},{"label":"Published posts only","code":"Post.published"},{"label":"Draft posts","code":"Post.drafts"},{"label":"First post with comments","code":"post = Post.first\nputs \"Post: #{post.title}\"\nputs \"Comments: #{post.comments.count}\"\npost.comments\n"},{"label":"Word count","code":"Post.all.map { |p| [p.title, p.word_count] }"},{"label":"Create a comment","code":"Post.first.comments.create(author: 'You', content: 'Hello from the REPL!')"}],"autorun":"Post.published.map(&:title)"}
</script>
<div class="playground-toolbar"><select class="playground-examples"><option value="">-- examples --</option></select></div>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Post</span><span class="p">.</span><span class="nf">published</span><span class="p">.</span><span class="nf">map</span><span class="p">(</span><span class="o">&amp;</span><span class="ss">:title</span><span class="p">)</span></code></pre></div></div>
<div class="playground-warning d-none">This does not play nicely with iOS.</div>
<div class="playground-controls">
<div class="playground-controls-buttons">
<button class="playground-run" disabled="">Run</button>
<button class="playground-reset" disabled="">Reset</button>
</div>
<span class="playground-controls-note">loads ~76MB Rails environment</span>
</div>
<div class="playground-output"><pre><code><span style="font-weight:700"><span style="color:#4ec9b0">Post Load (0.1ms)</span>  <span style="font-weight:700"><span style="color:#569cd6">SELECT "posts".* FROM "posts" WHERE "posts"."published" = 1</span>
=&gt; ["Getting Started with Ruby", "Rails is Magic"]</span></span></code></pre></div>
</div>

<h2 id="the-stack">The stack</h2>

<p>The playground runs <a href="https://github.com/palkan/wasmify-rails">wasmify-rails</a>, which compiles a Rails application to WebAssembly. The 76MB bundle includes Ruby 3.2 and SQLite—everything needed to run ActiveRecord queries in the browser. Each playground gets an in-memory SQLite database that lives only for the duration of your session.</p>

<p>The WASM bundle loads once and is shared across all playgrounds on a page. Initializing a Ruby VM is expensive—around 2-3 seconds on my ripping-fast machine, so spinning up separate VMs per playground would make the UX unusable. Instead, all playgrounds share a single VM but get isolated execution contexts and separate databases.</p>

<h2 id="vm-initialization">VM initialization</h2>

<p>When the first playground loads, it triggers VM initialization:</p>

<pre><code class="language-javascript">export async function initVM() {
  if (vm) return vm;

  await initSqlite();

  dbProxy = createDatabaseProxy();
  registerSQLiteWasmInterface(self, dbProxy);

  vm = await initRailsVM(`${window.CDN_HOST}/assets/rails-playground/app.wasm`, {
    database: { adapter: "sqlite3_wasm" },
  });

  vm.eval("REPL_WORKSPACES = {}");
  return vm;
}
</code></pre>

<p>The <code>registerSQLiteWasmInterface</code> call is where things get interesting. It wires up a JavaScript object to handle all SQLite operations from Ruby. Normally you’d pass a database instance directly, but I pass a proxy instead. More on that below.</p>

<p><code>REPL_WORKSPACES</code> is a Ruby hash that stores isolated workspace objects. Each playground gets its own entry—a plain <code>Object.new</code> that serves as the evaluation context.</p>

<h2 id="per-playground-database-isolation">Per-playground database isolation</h2>

<p>ActiveRecord assumes a single database connection. You call <code>establish_connection</code> once at boot and that connection handles all queries. But I wanted each playground to have its own isolated database. If you <code>Post.create</code> in playground A, it shouldn’t show up in playground B’s queries.</p>

<p>The obvious solutions don’t work well here. Running separate VMs would mean 76MB downloads and multi-second init times per playground. Using SQLite’s <code>ATTACH DATABASE</code> would require rewriting queries to prefix table names. Prefixing table names in the schema would break model definitions.</p>

<p>Rails has built-in multiple database support—you can use <code>connected_to</code> to switch databases at runtime. But it doesn’t help here. When you call <code>registerSQLiteWasmInterface(self, db)</code>, you’re registering a single JavaScript object that handles all SQLite operations from Ruby. The <code>sqlite3_wasm</code> adapter doesn’t use the <code>database</code> path from connection configs to open files—it just calls methods on whatever JavaScript object was registered. The path is ignored. So even if you tried <code>connected_to(database: { adapter: 'sqlite3_wasm', database: '/other.sqlite3' })</code>, you’d still hit the same database. The routing has to happen on the JavaScript side because that’s where the actual SQLite instances live.</p>

<p>I could wrong about all this, I don’t know. But it works! So I’m going to assume that I made some better choices.</p>

<p>The solution I landed on: a JavaScript Proxy that intercepts all database operations and routes them to different SQLite files based on which playground is currently executing.</p>

<pre><code class="language-javascript">const databases = new Map();
let currentWorkspace = null;

export const getOrCreateDatabase = async (workspaceId) =&gt; {
  await initSqlite();

  if (!databases.has(workspaceId)) {
    const db = new sqlite3.oo1.DB(':memory:', 'c');
    databases.set(workspaceId, db);
  }

  return databases.get(workspaceId);
};

export const createDatabaseProxy = () =&gt; {
  const handler = {
    get(target, prop) {
      const db = getCurrentDatabase();
      if (!db) throw new Error("no active database");
      const value = db[prop];
      if (typeof value === 'function') {
        return value.bind(db);
      }
      return value;
    }
  };

  return new Proxy({}, handler);
};
</code></pre>

<p>The proxy looks like a normal SQLite database object to the Ruby VM. When Ruby calls any method on it—<code>exec</code>, <code>prepare</code>, whatever—the proxy looks up the current workspace’s database from the Map and forwards the call there. The Ruby side has no idea this indirection exists.</p>

<p>Before each operation, we switch workspaces:</p>

<pre><code class="language-javascript">async function switchToWorkspace(workspaceId) {
  await getOrCreateDatabase(workspaceId);
  setCurrentWorkspace(workspaceId);

  vm.eval(`
    ActiveRecord::Base.connection.disconnect! rescue nil
    ActiveRecord::Base.establish_connection(adapter: 'sqlite3_wasm')
  `);
}
</code></pre>

<p>The <code>disconnect!</code> and <code>establish_connection</code> cycle forces ActiveRecord to drop its cached connection and create a new one. When it does, the <code>sqlite3_wasm</code> adapter calls back into JavaScript to get a database handle—and now the proxy routes to the new workspace’s database file. ActiveRecord thinks it’s talking to the same database it always was. It’s not.</p>

<p>This pattern—using a proxy to redirect operations based on runtime context—is useful whenever you need to multiplex a single-connection abstraction across multiple backends. I’ve used similar approaches for multi-tenant database routing in production Rails apps, though there you’d typically use <code>ActiveRecord::Base.connected_to</code> rather than raw connection cycling. Hopefully.</p>

<h2 id="per-playground-setup">Per-playground setup</h2>

<p>Each playground can have setup code that runs before the user’s code executes. This is useful for pre-configuring state without cluttering the editor.</p>

<p>Schema, models, and seed data run in the global context—model classes need to be globally accessible. But setup code runs inside the workspace’s <code>instance_eval</code>, so any instance variables it defines are scoped to that playground. You can set <code>@user = User.first</code> in setup and reference <code>@user</code> in the editor without leaking state to other playgrounds.</p>

<h2 id="code-evaluation">Code evaluation</h2>

<p>When you click Run, the code is wrapped and executed in the workspace context:</p>

<pre><code class="language-javascript">export async function evaluate(code, workspaceId) {
  await switchToWorkspace(workspaceId);

  const workspace = `REPL_WORKSPACES['${workspaceId}']`;

  const wrapped = `
    begin
      __repl_code__ = &lt;&lt;~'REPL_CODE'
${code}
REPL_CODE
      __repl_result__ = ${workspace}.instance_eval(__repl_code__)
      if __repl_result__.respond_to?(:to_a)
        { success: true, value: __repl_result__.to_a.map(&amp;:inspect).join("\\n") }
      else
        { success: true, value: __repl_result__.inspect }
      end
    rescue =&gt; e
      { success: false, error: e.class.name, message: e.message }
    end
  `;

  const result = vm.eval(wrapped);
  return result.toJS();
}
</code></pre>

<p>The code goes into a heredoc for the same injection-prevention reasons as setup. Then it’s <code>instance_eval</code>‘d on the workspace object. This means <code>self</code> inside the user’s code is that workspace object, and any instance variables are scoped to it. <code>@post</code> in playground A and <code>@post</code> in playground B are completely separate.</p>

<p>The result handling has a special case for objects that respond to <code>to_a</code>—typically ActiveRecord relations. Rather than inspecting the relation object itself (which would show something like <code>#&lt;ActiveRecord::Relation [...]&gt;</code>), we convert to an array and inspect each record on its own line. This makes the output much more readable when you’re querying multiple records.</p>

<h2 id="the-noscript-problem">The noscript problem</h2>

<p>The WASM approach works, but it has a fatal flaw: the playgrounds are invisible until JavaScript loads and executes. Disable JS, and you see nothing. This is bad for accessibility, bad for SEO, and bad for the subset of readers who browse with JS disabled.</p>

<p>The obvious fix is server-side rendering. But “server-side” for a static Jekyll site means build-time. I needed to run Rails code during <code>jekyll build</code> and inject the output into the HTML.</p>

<p>This turned a client-side-only feature into a progressive enhancement. With JS enabled, you get the full interactive experience. Without JS, you still see the code and its output—you just can’t modify and re-run it.</p>

<h2 id="build-time-rails-execution">Build-time Rails execution</h2>

<p>The build pipeline now includes a persistent Rails server that evaluates playground code during Jekyll’s render phase. Here’s why “persistent” matters: Rails takes several seconds to boot. If every playground block spawned a fresh Rails process, a post with 10 playgrounds would add minutes to build time.</p>

<p>Instead, a long-running Ruby process boots Rails once and handles requests over stdin/stdout:</p>

<pre><code class="language-ruby">require 'rails/all'

ENV['DATABASE_URL'] = 'sqlite3::memory:'

class App &lt; Rails::Application
  config.eager_load = false
end

App.initialize!

STDOUT.sync = true
puts "READY"

STDIN.each_line do |line|
  break if line.strip == "EXIT"
  config = JSON.parse(line)
  result = run_snippet(config)
  puts JSON.generate(result)
  puts "DONE"
end
</code></pre>

<p>The Jekyll plugin spawns this process once, sends each playground’s config as JSON, and reads back the results. The server stays warm across all posts in the build.</p>

<h2 id="isolated-execution-contexts">Isolated execution contexts</h2>

<p>Different blog posts define different schemas. A post about counter caches has different tables than a post about polymorphic associations. The build server needs to handle this without cross-contamination.</p>

<p>Each unique combination of schema, models, and seed data gets its own execution context:</p>

<pre><code class="language-ruby">$current_config_key = nil

def run_snippet(config)
  config_key = [config['schema'], config['models'], config['seed']].hash

  if $current_config_key != config_key
    # Tear down previous context
    ActiveRecord::Base.connection.tables.each do |table|
      ActiveRecord::Base.connection.drop_table(table, if_exists: true)
    end

    # Remove old model classes
    $user_constants.each do |const|
      Object.send(:remove_const, const) if Object.const_defined?(const)
    end

    # Build new context
    ActiveRecord::Schema.define { eval(config['schema']) }
    eval(config['models'])
    eval(config['seed'])

    $user_constants = Object.constants - $baseline_constants
    $current_config_key = config_key
  end

  # Execute the playground code
  eval(config['code'])
end
</code></pre>

<p>When the config changes, we drop all tables, remove all user-defined constants (model classes, modules, anything the previous context defined), and rebuild from scratch. When the config matches, we skip setup entirely—playgrounds within the same post share their schema and just run code.</p>

<p>This brought build time for a post with 10 playgrounds from around 17 seconds to under 2 seconds. The first playground pays the setup cost; the rest are nearly instant.</p>

<h2 id="capturing-interleaved-output">Capturing interleaved output</h2>

<p>Rails playground output has two streams: explicit output from <code>puts</code> statements, and implicit output from SQL query logging. These need to be interleaved correctly—if your code does <code>puts "before"</code>, runs a query, then <code>puts "after"</code>, the SQL should appear between them.</p>

<p>The WASM runtime handles this naturally because everything writes to the same output buffer. The build server needed more work.</p>

<p>First, we capture stdout to a tempfile during code execution:</p>

<pre><code class="language-ruby">tempfile = Tempfile.new('stdout')
original_stdout = STDOUT.dup
STDOUT.reopen(tempfile)
$capturing_sql = true

result = eval(config['code'])

$capturing_sql = false
STDOUT.reopen(original_stdout)

output = tempfile.read
</code></pre>

<p>Then we subscribe to ActiveRecord’s SQL notifications and write directly to the captured stdout:</p>

<pre><code class="language-ruby">ActiveSupport::Notifications.subscribe('sql.active_record') do |name, start, finish, id, payload|
  next unless $capturing_sql
  next if payload[:name] == "SCHEMA"

  # Interpolate bind parameters into the SQL
  sql = payload[:sql]
  type_casted = payload[:type_casted_binds] || []
  type_casted.each do |value|
    sql = sql.sub('?', ActiveRecord::Base.connection.quote(value))
  end

  duration = ((finish - start) * 1000).round(1)
  $stdout.puts "\e[1m\e[36m#{payload[:name]} (#{duration}ms)\e[0m  \e[1m\e[34m#{sql}\e[0m"
end
</code></pre>

<p>Because the notification handler writes to <code>$stdout</code> (which is redirected to the tempfile), SQL logs appear inline with <code>puts</code> output in the correct order.</p>

<h2 id="ansi-to-html-conversion">ANSI to HTML conversion</h2>

<p>The SQL logs use ANSI escape codes for colorization: cyan for the query name, blue for the SQL. The WASM runtime converts these to HTML spans with inline styles. The build server output needs to match exactly, or you’d see a visual flash when JS hydrates the page.</p>

<p>The conversion handles nested escape sequences and tracks open spans to ensure valid HTML:</p>

<pre><code class="language-ruby">ANSI_COLORS = {
  '30' =&gt; '#000', '31' =&gt; '#f14c4c', '32' =&gt; '#89d185', '33' =&gt; '#dcdcaa',
  '34' =&gt; '#569cd6', '35' =&gt; '#c586c0', '36' =&gt; '#4ec9b0', '37' =&gt; '#d4d4d4'
}.freeze

def ansi_to_html(str)
  escaped = str.gsub('&amp;', '&amp;amp;').gsub('&lt;', '&amp;lt;').gsub('&gt;', '&amp;gt;')
  out = String.new
  open_spans = 0
  scanner = StringScanner.new(escaped)

  until scanner.eos?
    if scanner.scan(/\e\[([0-9;]+)m/)
      codes = scanner[1].split(';')
      if codes.include?('0')
        out &lt;&lt; '&lt;/span&gt;' if open_spans &gt; 0
        open_spans = [open_spans - 1, 0].max
      else
        styles = []
        codes.each do |c|
          styles &lt;&lt; 'font-weight:700' if c == '1'
          styles &lt;&lt; "color:#{ANSI_COLORS[c]}" if ANSI_COLORS[c]
        end
        if styles.any?
          open_spans += 1
          out &lt;&lt; %(&lt;span style="#{styles.join(';')}"&gt;)
        end
      end
    else
      out &lt;&lt; scanner.getch
    end
  end

  out &lt;&lt; ('&lt;/span&gt;' * open_spans)
  out
end
</code></pre>

<p>The WASM runtime uses separate ANSI codes for bold and color (<code>\e[1m\e[36m</code>), so the build server does too. Using combined codes (<code>\e[1;36m</code>) would produce different HTML structure—functionally equivalent but visually detectable during hydration.</p>

<h2 id="progressive-enhancement">Progressive enhancement</h2>

<p>The Jekyll plugin renders the full playground UI server-side: the code editor, the Run/Reset buttons, the output panel. JavaScript isn’t needed to display anything.</p>

<pre><code class="language-ruby">&lt;&lt;~HTML
&lt;div class="playground" data-runtime="rails"&gt;
  &lt;div class="playground-toolbar" hidden&gt;...&lt;/div&gt;
  &lt;div class="language-ruby highlighter-rouge"&gt;#{highlighted_code}&lt;/div&gt;
  &lt;div class="playground-controls"&gt;
    &lt;button class="playground-run" disabled&gt;Run&lt;/button&gt;
    &lt;button class="playground-reset" disabled&gt;Reset&lt;/button&gt;
  &lt;/div&gt;
  &lt;div class="playground-output"&gt;&lt;pre&gt;&lt;code&gt;#{ansi_to_html(output)}&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;/div&gt;
HTML
</code></pre>

<p>When JS loads, it checks if the controls already exist before injecting them:</p>

<pre><code class="language-javascript">function injectPlaygroundControls(el, runtime) {
  if (el.querySelector('.playground-controls')) {
    return true; // Already rendered server-side
  }
  // ... inject controls dynamically
}
</code></pre>

<p>For the output, JS checks if the HTML differs from plain text. If it does, the content was pre-rendered and doesn’t need transformation:</p>

<pre><code class="language-javascript">if (output &amp;&amp; output.textContent.trim()) {
  if (output.innerHTML === output.textContent) {
    output.innerHTML = transformer(output.textContent);
  }
}
</code></pre>

<p>Users with JS disabled see the playground with pre-computed output. Users with JS enabled can modify the code and re-run it. Same HTML serves both.</p>

<h2 id="noscript-styling">Noscript styling</h2>

<p>Interactive controls are hidden when JS is disabled:</p>

<pre><code class="language-html">&lt;noscript&gt;
  &lt;style&gt;
    .playground-controls,
    .playground-toolbar,
    .code-editor-input { display: none !important; }
  &lt;/style&gt;
&lt;/noscript&gt;
</code></pre>

<p>The output panel stays visible. The textarea overlay on the code editor hides, leaving just the syntax-highlighted code (which is still selectable and copyable).</p>

<h2 id="limitations">Limitations</h2>

<p>You can’t make HTTP requests or call external APIs from Ruby code. The WASM sandbox prevents it.</p>

<p>Gems with C extensions won’t compile to WASM. Pure Ruby gems work fine.</p>

<p>The full Ruby/Rails runtime is 76MB. It’s cached after the first load, but the initial download is noticeable on slow connections.</p>

<p>Build-time rendering adds complexity. The Jekyll plugin, Rails server script, and ANSI conversion all need to stay in sync with the client-side runtime. When I change how the WASM runtime formats output, I need to update the build pipeline to match.</p>

<p>For interactive demos and tutorials, these tradeoffs are worth it. You get real ActiveRecord with associations, scopes, validations, callbacks—everything you’d use in a typical Rails app. And now it works without JavaScript too.</p>
]]></content>
    
    <summary>I built a WASM-compatible Rails playground that runs entirely in your browser. No server, no setup, just Rails. This is built on the backs of people much smarter than me.
</summary>
    
    
    <category term="ruby"/>
    
  </entry>
  
  <entry>
    <title>How to get a job as a felon</title>
    <link href="/posts/how-to-get-a-job-as-a-felon/" rel="alternate" type="text/html"/>
    <id>/posts/how-to-get-a-job-as-a-felon/</id>
    <published>2025-11-20T00:00:00+00:00</published>
    <updated>2025-11-20T00:00:00+00:00</updated>
    <content type="html"><![CDATA[<p>Hi, I’m Josh. I was inmate number 71690-509. I went to federal prison. I got a job after. Here’s what I observed.</p>

<p>This isn’t motivational. I’m not going to tell you to “stay positive” or that “everything happens for a reason.” If you’re reading this, you’ve probably already waded through that garbage. You want to know what actually works.</p>

<p>Fair warning: my crime was running a sports streaming website. That’s a “cool” crime in comparison to many. People hear it and go “oh, what?!” rather than recoiling. If you’re reading this with an assault conviction or something involving theft, your mileage will vary. I’ll get into that.</p>

<p>Also: this is my experience. One data point. Take what’s useful.</p>

<p>One more thing: this is for people who have marketable skills but don’t know how to navigate the job search with a record. If you can do something valuable—code, weld, design, manage projects, whatever—and your problem is figuring out how to get someone to give you a shot despite your background, this is for you. If you don’t have skills yet, the calculus is different. You might need to build those first.</p>

<p>Wait, no, one more: my email is open; it’s hi@josh.mn.</p>

<hr />

<h2 id="disclose-early-disclose-to-the-decision-maker">Disclose early, disclose to the decision maker</h2>

<p>Here’s the thing I noticed: the longer you wait to tell someone, the worse it goes.</p>

<p>The conventional wisdom is to wait until they like you. Get your foot in the door, impress them in the interview, build rapport—then drop it. The theory is that by then they’re invested in you, so nothing bad can happen.</p>

<p>No, surprises are bad. Very bad. Waiting creates an awkward reveal. It feels like you were hiding something. Because you were.</p>

<p>My approach: I told them before we even got to the interview stage. For small companies, I’d skip the normal application process entirely. I’d find the decision maker—usually the founder or hiring manager—and send my resume directly.</p>

<p>Finding that person isn’t hard. LinkedIn shows you who works where and what their title is. Google “[company name] founder” or “[company name] engineering manager.” Once you have a name, figure out the email format—tools like <a href="https://hunter.io">Hunter</a> or <a href="https://rocketreach.co">RocketReach</a> will tell you whether a company uses firstname@company.com or firstname.lastname@company.com. Then you send the email.</p>

<p>The language I used: “Hey, I wanted to send over my resume. I just got out of federal prison—here’s some context.”</p>

<p>That’s it. No long explanation. No apologizing. Just the fact, stated plainly.</p>

<p>If I got to an interview through normal channels, I’d drop it at the end of the first conversation. Not at the beginning—you still want to have an actual conversation about the role. But before we wrapped up, I’d say something like: “Hey, just so you know, I can’t pass a background check.”</p>

<p>The logic: if the interviewer isn’t a decision maker, I’d rather they pass along “they were cool, also they have a record” instead of “they have a record.” Sequence matters.</p>

<p>Most people’s next question is “what happened?” Keep that answer to one sentence. Mine was: “I had a sports streaming website.” Enough to satisfy curiosity, not so much that you’re arguing your case. If they inquired further, I never felt the need to explain every detail.</p>

<p>What I noticed: nobody reacted badly. Not once. Part of that is probably the nature of my conviction. But I also think being direct about it signals something. It says you’re not going to be weird about this, that you’ve processed it, that it’s not a landmine they have to tiptoe around.</p>

<p>The people who would’ve had a problem with it? They filtered themselves out before we wasted each other’s time.</p>

<hr />

<h2 id="target-small-companies">Target small companies</h2>

<p>I aimed for companies under 50 employees.</p>

<p>It’s not that larger companies won’t hire felons. Some will. But the process is different. Large companies have HR departments with checklists. They have legal teams that get nervous. They have policies written by people who will never meet you.</p>

<p>Small companies have humans making decisions.</p>

<p>At a small company, the person reading your resume might be the same person who’ll be working next to you. They’re evaluating whether you can do the job and whether they want to be in a room with you. A conviction is a data point, not an automatic disqualification.</p>

<p>There’s also less bureaucracy to navigate. No applicant tracking systems filtering you out before a human sees your name. No mandatory background check policies that exist because someone in legal wrote them five years ago.</p>

<p>I don’t know the exact threshold, but somewhere around 50 employees is where things start getting more formal. Below that, you’re more likely to be talking to the person who actually makes the call.</p>

<hr />

<h2 id="consider-remote-work">Consider remote work</h2>

<p>Remote work changes the dynamics.</p>

<p>You’re not in an office where people are going to see you every day and wonder about your background. The relationship is more transactional—can you do the work? Are you responsive? Do you deliver?</p>

<p>It also widens your options. You’re not limited to companies in your city. A startup in Austin or a company in New York might not care about your record—and you don’t have to relocate to find out.</p>

<p>Remote roles tend to be more output-focused. The thing that matters is whether you shipped the thing, not whether you have good office presence. For someone with a record, that’s an easier evaluation to win.</p>

<hr />

<h2 id="contract-and-freelance-work">Contract and freelance work</h2>

<p>Contract and freelance work is probably easier than full-time employment.</p>

<p>The commitment is lower on both sides. A company hiring a contractor isn’t thinking “is this person going to be here in five years?” They’re thinking “can this person finish this project?” That’s a simpler question to answer yes to.</p>

<p>Background checks are also less common for contract work. Not universal, but less common. Companies bring on contractors all the time without running them through the same process they’d use for a full-time hire.</p>

<p>It’s also a way to build a track record. Do good work for a few clients, get some references, demonstrate you’re reliable—then convert that into full-time if you want. Or don’t. Some people prefer staying independent.</p>

<p>I went direct to full-time, but if I’d had more trouble, this is probably the route I would’ve tried. But, the conditions of my supervised release were to maintain employment; I’m not sure if contract or freelance would have been an issue, or just a point to run by my probation officer.</p>

<hr />

<h2 id="own-it-on-paper">Own it on paper</h2>

<p>I put my incarceration on my resume. I listed myself as an inmate. I included my inmate number.</p>

<p>This sounds insane. I know.</p>

<p>Here’s the logic: I wanted to get there first. I didn’t want someone to Google me, find out, and wonder why I didn’t mention it. I wanted to control the narrative.</p>

<p>Under my work history, there’s a gap. In that gap, it says I was an inmate in federal prison. The description says something like “this explains my employment gap, right?”</p>

<p>Same thing on <a href="https://linkedin.com/in/joshmn">LinkedIn</a>. It’s there, in my profile, for anyone to see. It’s been described as <a href="https://x.com/jrobah/status/1986759762449453527">cracked</a>.</p>

<p>You might think this would cost me opportunities. Maybe it has. But here’s what I noticed: the opportunities I got were with people who already knew and didn’t care. No awkward reveal. No “oh, I didn’t realize…” conversation three weeks into the job. They hired me knowing exactly what they were getting.</p>

<p>In hindsight, I probably should have included a link to my actual site. That would’ve been even more useful context.</p>

<hr />

<h2 id="the-gap-question">The gap question</h2>

<p>If someone asks what you did during your incarceration, the answer is: prison.</p>

<p>That’s it. You were in prison.</p>

<p>No system in the US is designed to teach you a marketable skill while you’re inside. The programs are mostly a joke. If you happened to learn something useful, great—mention it. But don’t feel like you need to justify that time or prove you were “productive.” You were incarcerated. That’s the explanation.</p>

<p>What matters more is what you did before and what you can do now. The gap is just context for why you weren’t working.</p>

<hr />

<h2 id="what-employers-actually-care-about">What employers actually care about</h2>

<p>They care whether you can do the work, and if you’re good to work with.</p>

<p>I’m a software engineer. I have skills. When I talked to potential employers, they wanted to know if I could build things. The conviction was a thing they had to think about, but it wasn’t the thing.</p>

<p>What I noticed: the employers who hesitated weren’t people I wanted to work with anyway.</p>

<p>This sounds like cope, but hear me out. Trust is foundational to any relationship. If someone’s default is suspicion—if their first instinct is to see you as a risk to be managed rather than a person to be evaluated—that’s going to color everything. Every mistake you make will be viewed through a different lens. Every time something goes wrong, there’s going to be a voice in the back of their head.</p>

<p>The people who default to trust? They’re better to work with anyway. For anyone. The conviction just surfaces this faster.</p>

<p>If you’ve been locked up for a while, you’re also dealing with a lot at once. New routine. Recalibrating to normal life. Society is generally willing to acknowledge this. But you have to advocate for yourself. You have to be honest about where you are. People will usually give grace, and (hopefully) additional resources—time, attention, understanding.</p>

<p>Being incarcerated can actually be useful context for an employer. It explains things. It gives them a framework for understanding why you might need a little flexibility, why certain things might take you a minute to adjust to. Most decent people will give you that space—if you’re upfront about needing it.</p>

<hr />

<h2 id="background-check-basics">Background check basics</h2>

<p>Here’s the mechanical stuff, since you might be wondering.</p>

<p>Background checks typically show convictions, not arrests. How far back depends on the state—some have a seven-year limit, others don’t. Federal convictions generally don’t have a time limit on reporting.</p>

<p>Under the Fair Credit Reporting Act (FCRA), if an employer uses a third-party background check and decides not to hire you based on what they find, they have to give you a copy of the report and let you dispute any inaccuracies. This matters if there’s something wrong on there. It happens.</p>

<p>Ban the box laws exist in a lot of places now. The idea is that employers can’t ask about criminal history on the initial application—they have to wait until later in the process. Whether this actually helps is debatable. It delays the question; it doesn’t eliminate it. And plenty of employers still ask, law or not.</p>

<p>For me, ban the box was mostly irrelevant because I was disclosing upfront anyway. I wasn’t waiting for them to ask.</p>

<p>If an application has that checkbox—”have you ever been convicted of a felony?”—I’d check yes. But I’d also try to get to the decision maker directly with context, so they weren’t just seeing a checked box with no explanation.</p>

<hr />

<h2 id="ignore-most-re-entry-advice">Ignore most re-entry advice</h2>

<p>The re-entry staff told me I needed to get a job at McDonald’s.</p>

<p>Their reasoning: they didn’t think I’d be able to get a job with a computer crime on my record.</p>

<p>This is degrading. It’s also wildly out of touch.</p>

<p>Even the officers at the prison were like “oh, that’s marketable.” They understood the market better than the re-entry people.</p>

<p>The re-entry system is built for people without resources, without skills without networks. If that’s you, I hope it was useful to you. But if you’re reading this—if you’re the kind of person who Googles for real information instead of accepting what you’re handed—you’re probably not the person that advice was designed for.</p>

<p>Most of the re-entry material assumes you’re useless. It assumes you have no options. If you have marketable skills, if you’re resourceful, if you’re capable of reaching out directly to people and making a case for yourself—ignore their “playbook.” It wasn’t written for you.</p>

<hr />

<h2 id="if-you-keep-getting-rejected">If you keep getting rejected</h2>

<p>I didn’t have problems finding work. But I had skills, a “cool” crime, and some luck. Not everyone has that combination.</p>

<p>If you’re getting rejected repeatedly, a few things to consider:</p>

<p>The pool might be wrong. If you’re applying to big companies with formal HR processes, you’re fighting the system. Switch to smaller companies where you can talk to humans.</p>

<p>The timing might be off. If you’re disclosing too late, it feels like deception. If you’re disclosing too early—like in the first sentence of a cold email with no context about who you are—it might be the only thing they register. Find the balance.</p>

<p>The framing might need work. How you talk about it matters. If you’re apologetic, defensive, or clearly anxious, that energy transfers. Practice saying it plainly until it feels routine.</p>

<p>The skills might not be there yet. This is harder to hear, but if you don’t have something to offer that outweighs the risk someone perceives, you’re going to struggle. That doesn’t mean give up—it means figure out what skill you can build that makes you valuable enough that the conviction becomes a footnote.</p>

<p>And sometimes it’s just numbers. You might have to talk to more people than someone with a clean record. That’s not fair, but it’s real. Keep going.</p>

<p>I’m happy to talk to you—or anyone—about any of this.</p>

<hr />

<h2 id="telling-coworkers">Telling coworkers</h2>

<p>Be open about it.</p>

<p>In my industry, people Google their colleagues. Yours probably do too. If your record is findable—and it probably is—someone’s going to find it. Better they hear it from you than discover it and wonder why you never mentioned it.</p>

<p>I told my whole company at the first all-hands I was in. That’s not the only way to do it, but the principle is the same: own your story, be your own narrator. If you let other people tell it, you don’t control how it sounds.</p>

<p>This doesn’t mean you need to announce it on your first day. But don’t hide it either. If it comes up, say it plainly. If someone asks where you worked before, and the answer is nowhere because you were incarcerated, say that. Most people will move on faster than you expect.</p>

<p>The people who can’t handle it were going to be weird about it regardless. Better to find out early.</p>

<hr />

<h2 id="owning-your-story">Owning your story</h2>

<p>Here’s something that happened after I got hired at my current job.</p>

<p>The entire company was in a compliance meeting with the company lawyer. I was there too—flown in, I was the only remote employee. New face. Hadn’t met most people yet.</p>

<p>Turns out the lawyer does both corporate law and federal criminal defense. She didn’t know I was in the meeting. At some point she jokingly said something like “and nobody here would do anything to go to prison, so…”</p>

<p>I said, “Hey Kelly.”</p>

<p>She looked at me, confused.</p>

<p>I said my name. She smirked.</p>

<p>Then I announced to the room, “Oh yeah, I just got out of federal prison. Tell you guys later.”</p>

<p>And I did—at the all-hands. It was well-received.</p>

<p>I think owning your story matters. If you treat it like a shameful secret, people pick up on that energy. If you treat it like a thing that happened to you, that you’ve dealt with, that’s part of your history but doesn’t define you—people pick up on that too.</p>

<p>My employer has since mentioned to the team that my background was a factor they considered when hiring—and that it’s been a success. That’s only possible because I was upfront. No surprises. No awkward discoveries. Just a fact on the table from day one.</p>

<hr />

<h2 id="what-this-doesnt-cover">What this doesn’t cover</h2>

<p>Different crimes land differently.</p>

<p>I have a “cool” crime. Sports streaming. People hear that and they’re intrigued, not alarmed.</p>

<p>Drug dealing convictions can be reframed. You were in sales. Technically true, and it lands better than “I sold drugs.” You managed inventory, handled customer relationships, dealt with cash flow. The framing matters.</p>

<p>Assault makes people uncomfortable. They’re going to look at you differently. That’s reality.</p>

<p>Theft will preclude you from certain positions. You’re not going to be handling money. Possibly not handling inventory. That’s just how it is.</p>

<p>There are crimes I’m not going to address here. You know which ones. If your conviction involves certain categories of harm, this advice doesn’t apply and I’m not going to pretend it does.</p>

<p>Time matters too. My older boring white-collar state crimes are more than 15 years back—that makes it essentially irrelevant. If you’re recently convicted of something serious, the calculus is different.</p>

<hr />

<h2 id="the-psychological-part">The psychological part</h2>

<p>The hardest part isn’t logistical.</p>

<p>It’s understanding that society can accept you. That you’re not the worst person in the world because you went to prison. That the narrative in your head—where everyone is judging you, where every door is closed, where you’re permanently marked—isn’t accurate.</p>

<p>I’m humbled by the experience. I’d be lying if I said otherwise. But I don’t let it define me.</p>

<p>There’s a difference between being desperate and being humbled. Desperate makes you accept things you shouldn’t accept. It makes you undersell yourself. It makes you grateful for scraps.</p>

<p>Humbled means you understand what happened, you’ve integrated it, and you’re moving forward. You don’t carry it like a weight you’re dragging. You carry it like something you’ve processed.</p>

<p>It gets easier over time—emotionally, mentally. The first conversation where you disclose is the hardest. The twentieth is routine.</p>

<hr />

<h2 id="the-summary">The summary</h2>

<p>Disclose early. Disclose to the person who actually makes the decision. Use plain language—”I can’t pass a background check”—and give a one-sentence explanation if they ask.</p>

<p>Target small companies. Under 50 employees. Talk to humans, not systems.</p>

<p>Own it on paper. Put it on your resume, put it on LinkedIn. Get there first.</p>

<p>Care about whether they’re the kind of people who default to trust. The ones who don’t aren’t worth your time anyway.</p>

<p>Ignore advice designed for people with no options. If you have skills, use them.</p>

<p>You’re just as good as everyone else. Act like it.</p>
]]></content>
    
    <summary>Hi, I’m Josh. I was inmate number 71690-509. I went to federal prison. I got a job after. Here’s what I observed.
</summary>
    
    
    <category term="personal"/>
    
  </entry>
  
  <entry>
    <title>When you don&apos;t look autistic</title>
    <link href="/posts/when-you-dont-look-autistic/" rel="alternate" type="text/html"/>
    <id>/posts/when-you-dont-look-autistic/</id>
    <published>2025-10-14T00:00:00+00:00</published>
    <updated>2025-10-14T00:00:00+00:00</updated>
    <content type="html"><![CDATA[<p>I get this a lot. “You don’t look autistic.” It’s meant as a compliment, I think. Like I’ve done a good job of hiding it. Gold star for passing. Or it’s a remark that implies I don’t have autism because I don’t visibly struggle.</p>

<p>You know, like you can’t have COVID if you don’t get tested. Or you don’t look like you have cancer—is that one?</p>

<p>What I want to ask—but don’t, because that would be rude and I’ve learned not to be rude in ways that upset people—is: what does autistic look like?</p>

<h2 id="what-they-expect">What they expect</h2>

<p>Based on the reactions I get, autistic apparently looks like:</p>

<ul>
  <li>A white male child (I am two of these things, unfortunately);</li>
  <li>Obsessed with trains or math;</li>
  <li>Unable to make eye contact ever, under any circumstances;</li>
  <li>Completely incapable of sarcasm;</li>
  <li>Some degree of intellectual disability;</li>
  <li>Rain Man, basically (also a Minnesotan) (I am one of those things) <!-- style:ignore --></li>
</ul>

<p>Here’s the thing: I am obsessed with trains. I will go out of my way to get stopped at a railroad crossing. I love algebra. So the stereotypes aren’t entirely wrong—they’re just incomplete. Matching a few boxes doesn’t mean you match all of them, and not matching the visible ones doesn’t mean you’re not autistic.</p>

<p>The eye contact thing is partially true. I can do it in short painful bursts when I force myself. The rest of the time I’m looking for spaceships—staring past people, through them, at something over their shoulder, maybe up in the air. It’s not that I can’t make eye contact. It’s that it costs something every time I do; it’s manual labor.</p>

<p>I can use sarcasm. But my voice is flat. My tone—if I’m not consciously thinking about it—doesn’t shift the way it’s supposed to. So I’ll say something sarcastic and people take it literally, and then I have to explain that it was a joke, and nothing kills a joke faster than having to explain it was a joke.</p>

<h2 id="how-we-got-here">How we got here</h2>

<p>The image of autism in most people’s heads didn’t come from nowhere. It was constructed, historically, by who got studied and who got diagnosed.</p>

<p>In the 1940s, Leo Kanner published his observations of children with what he called “<a href="https://embryo.asu.edu/pages/autistic-disturbances-affective-contact-1943-leo-kanner">early infantile autism</a>.” The children he studied were, by and large, the ones who couldn’t hide it—kids with obvious social differences, limited speech, visible distress. Kanner’s work became the foundation for <a href="https://pmc.ncbi.nlm.nih.gov/articles/PMC8531066/">how autism was understood for decades</a>: a severe childhood condition, unmistakable, tragic.</p>

<p>Around the same time, Hans Asperger was studying a different group of kids in Vienna—children who were socially odd but highly verbal, often with intense intellectual interests. His work was in German, during World War II, and it <a href="https://en.wikipedia.org/wiki/Hans_Asperger">didn’t make it into the English-speaking medical mainstream until the 1980s</a>. For forty years, the only autism that existed in the diagnostic manuals was Kanner’s version. The visible kind. The kind you couldn’t miss.</p>

<p>This is why “you don’t look autistic” is even a sentence. And fuck I hate that it is. The diagnosis was built around people who couldn’t mask, couldn’t pass, couldn’t perform normalcy. Everyone else—those who learned to fake it, who white-knuckled their way through social situations, who figured out the rules through painful trial and error—didn’t count. They weren’t autistic. They were just weird, or difficult, or anxious, or “quirky.”</p>

<p>The DSM didn’t <a href="https://en.wikipedia.org/wiki/History_of_autism">merge Asperger’s syndrome into the autism spectrum until 2013</a> with its release of the DSM-5. That’s not ancient history. That’s recent. A whole generation of autistic people grew up being told they weren’t autistic because they didn’t match a profile designed around the most visible cases. Or they had “Asperger’s” because they didn’t fit the prototypical profile, as defined in the DSM-IV-TR from 2000.</p>

<p>Steve Silberman’s <em><a href="https://bookshop.org/p/books/neurotribes-the-legacy-of-autism-and-the-future-of-neurodiversity-steve-silberman/8503640">NeuroTribes</a></em> traces this history in detail—how autism was defined, redefined, narrowed, and eventually expanded. The short version: what “looks autistic” is an artifact of diagnostic criteria that excluded most autistic people for most of the diagnosis’s existence. The image in your head is a historical accident, not a medical reality.</p>

<h2 id="the-misdiagnosis-pipeline">The misdiagnosis pipeline</h2>

<p>When you don’t “look autistic,” it’s incredibly difficult to get diagnosed with autism. You get diagnosed with everything else instead.</p>

<p>Depression. Anxiety. Bipolar disorder. Borderline personality disorder if you’re seriously unlucky. OCD. ADHD. Eating disorders. Social anxiety disorder. Sometimes several of these at once, stacked on top of each other like a diagnostic Jenga tower that never quite stabilizes.</p>

<p>My social uncertainty was an issue solvable by SSRIs and SNRIs. My unwillingness to go outside, my saying-dumb-things disorder was chalked up to anxiety and an impulse disorder.</p>

<p>Some of these aren’t exactly wrong. Autistic people often do have depression and anxiety—usually because existing in a world that doesn’t fit your brain is depressing and anxiety-inducing. But they’re downstream diagnoses. They’re treating symptoms while missing the cause.</p>

<p>One of the problems with autistic people with low-support needs is that the script to pass—call it an acceptance test, since this is a technical blog sometimes—is rehearsed over and over. Short of a complete biography, short of seeing an autistic person with low-support needs in the wild, it’s hard to understand the problems faced because the mask can be worn.</p>

<p>If you’re a female with autism, doubly-so. You are (usually) explicitly taught manners, how to be polite, and there are scripts associated with it, installed at a young age.</p>

<p>Here’s the thing about prevalence: <a href="https://www.nimh.nih.gov/health/statistics/autism-spectrum-disorder-asd">autism occurs at roughly the same rate</a> across sexes, races, and cultures. The <a href="https://www.cdc.gov/autism/data-research/index.html">CDC estimates 1 in 31 children</a> in the US; the <a href="https://www.who.int/news-room/fact-sheets/detail/autism-spectrum-disorders">WHO says 1 in 100 globally</a>. The <a href="https://publications.aap.org/aapnews/news/31909/CDC-report-Autism-rate-rises-to-1-in-31-children">variation you see across countries</a>—California at 3.9%, France at 0.36%—isn’t biological. It’s access. It’s awareness. It’s whether your culture even has a framework for recognizing autism in the first place.</p>

<p>The famous 4:1 male-to-female ratio? That’s <a href="https://pubmed.ncbi.nlm.nih.gov/28545751/">another diagnostic artifact</a>, not a biological reality. When researchers adjust how they evaluate participants, the ratio drops. <a href="https://pubmed.ncbi.nlm.nih.gov/28545751/">One study brought it from 4.2 to 3.3</a> just by changing evaluation methods. A <a href="https://www.thetransmitter.org/spectrum/autisms-sex-ratio-explained/">Danish study tracked diagnoses over time</a> and watched the ratio fall from 8:1 in 1995 to 3:1 within fifteen years. The gap is closing not because more women are becoming autistic, but because we’re finally getting better at recognizing autism that doesn’t look like a white boy who likes trains.</p>

<p>Same pattern with race and ethnicity. In the US, <a href="https://www.nimh.nih.gov/health/statistics/autism-spectrum-disorder-asd">autism prevalence is now similar across racial groups</a>—and in some data, higher among Asian, Hispanic, and Black children than white children. This isn’t because autism suddenly became more common in minority communities or PFAs. It’s because outreach, screening, and destigmatization finally reached them. The autism was always there. The diagnoses weren’t.</p>

<p>What I’m trying to say is that the little girl in the tribe that hasn’t been contacted by civilization is just as likely to have autism as the girl next to the 3M factory in Maplewood, Minnesota.</p>

<p><em><a href="https://bookshop.org/p/books/unmasking-autism-discovering-the-new-faces-of-neurodiversity-devon-price/17363182">Unmasking Autism</a></em> talks about this extensively. The average age of autism diagnosis for people who can mask is years or decades later than for those who can’t. <a href="https://pubmed.ncbi.nlm.nih.gov/38285291/">Women get diagnosed later than men</a>, on average, because the diagnostic criteria were built around how autism presents in boys. People of color get diagnosed later or not at all, because the research was conducted almost entirely on white kids.</p>

<p>If you’re good at masking—if you’ve learned to perform normalcy well enough that clinicians don’t see the autism—you get funneled into the wrong diagnostic categories. You spend years on medications that don’t quite work, in therapy that helps but doesn’t address the underlying thing, wondering why you’re still struggling when you’re doing everything right.</p>

<p>Quick note on therapy: autistic people often can’t identify what they struggle with, so they don’t bring it up. If a therapist asks specific questions, they can get into detail. But until someone probes, what’s normal for the autistic person will be treated as normal—not flagged as a sign of something else.</p>

<p>I collected diagnoses for years before anyone said the word “autism.” Each one explained just part of the picture. None of them explained all of it. It wasn’t until I read about autism in adults—specifically adults who’d learned to mask—that everything clicked into place.</p>

<p>The “you don’t look autistic” thing isn’t just fucking annoying. It’s a diagnostic barrier. It’s the reason people spend decades thinking they’re broken in fifteen different ways instead of understanding they’re autistic in one.</p>

<p>And the stakes are high. <a href="https://molecularautism.biomedcentral.com/articles/10.1186/s13229-018-0226-4">Autistic people are significantly more likely to die by suicide</a> than the general population—some studies suggest <a href="https://www.healthdata.org/research-analysis/library/global-burden-suicide-mortality-among-people-autism-spectrum">nine times more likely</a> than their neurotypical peers. <a href="https://pubmed.ncbi.nlm.nih.gov/37668055/">Suicidal ideation is even more common</a>. The numbers are so much worse for late-diagnosed adults, for women, for anyone who spent years masking without knowing why they had to.</p>

<p>The stress of adulthood builds up. A <a href="https://pmc.ncbi.nlm.nih.gov/articles/PMC10018918/">2023 meta-analysis</a> found that 34% of autistic adults without intellectual disability experience suicidal ideation, and 24% have attempted suicide. <a href="https://molecularautism.biomedcentral.com/articles/10.1186/s13229-018-0226-4">72% surpass clinical thresholds for suicide risk</a>. These aren’t people with visible, profound autism who might receive support and accommodation. These are the ones who mask, who pass, who look fine. The <a href="https://molecularautism.biomedcentral.com/articles/10.1186/s13229-018-0226-4">camouflaging itself is a risk factor</a>—research shows it <a href="https://molecularautism.biomedcentral.com/articles/10.1186/s13229-018-0226-4">independently predicts suicidal thoughts</a> even after controlling for depression and anxiety. You spend decades performing normalcy, burning through energy you don’t have, and the debt compounds. <a href="https://www.cambridge.org/core/journals/psychological-medicine/article/anxiety-and-depression-in-adults-with-autism-spectrum-disorder-a-systematic-review-and-metaanalysis/CDC4FF29C3DC504768E375EE65019E0C">Depression hits autistic people at four times the general rate</a>. <a href="https://www.cambridge.org/core/journals/psychological-medicine/article/anxiety-and-depression-in-adults-with-autism-spectrum-disorder-a-systematic-review-and-metaanalysis/CDC4FF29C3DC504768E375EE65019E0C">Anxiety affects 50% of autistic adults</a>. <a href="https://molecularautism.biomedcentral.com/articles/10.1186/s13229-018-0226-4">Late diagnosis correlates with higher suicidality</a>. The pattern is clear: the longer you go without knowing why you’re struggling, the worse it gets.</p>

<p>This isn’t an abstract statistic to me. I’ve been in dark places. The kind where you’re not sure you want to keep doing this, whatever “this” is. Knowing I was autistic didn’t fix everything, but it gave me a framework. It turned “I’m fundamentally broken” into “I’m running different software in a world designed for a different operating system.” That reframe matters. It’s the difference between hating yourself and understanding yourself.</p>

<p>When someone doesn’t “look autistic” and doesn’t get diagnosed, they don’t get that framework. They just get the self-hatred. And <a href="https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2774853">sometimes that’s fatal</a>.</p>

<h2 id="the-retardation-thing">The retardation thing</h2>

<p>Let’s just say it: when most people hear “autistic,” they picture someone with an intellectual disability. The image in their head is someone who can’t live independently, can’t hold a job, can’t have a conversation.</p>

<p>Autism and intellectual disability are different things. They can overlap, but they usually don’t. <a href="https://www.cdc.gov/autism/data-research/index.html">The majority of autistic people have average or above-average intelligence</a>. The struggles are social, sensory, and communicative—not cognitive.</p>

<p>But the conflation persists, partly because of that same history. The autistic people who got studied, diagnosed, and institutionalized were often the ones with co-occurring intellectual disabilities. They were the most visible. The ones who could pass—who could mask, who could perform—didn’t end up in the studies. So the studies said autism looked like intellectual disability, because those were the only autistic people anyone was counting.</p>

<p>This is why “you don’t look autistic” lands weird. It’s not really about looking autistic. It’s about not looking intellectually disabled. Which I’m not. And neither are most autistic people. But thanks for the accidental insult wrapped in a compliment.</p>

<h2 id="were-all-a-little-autistic">“We’re all a little autistic”</h2>

<p>This one makes me want to walk into the sea.</p>

<p>When I tell someone I’m autistic, sometimes they respond with: “Oh, I think everyone’s a little autistic. I’m so OCD about my desk!” Or: “I get overwhelmed at parties too, we’re all on the spectrum somewhere.”</p>

<p>No. No, we’re not.</p>

<p>Everyone has moments of social awkwardness. Everyone gets overwhelmed sometimes. Everyone has preferences and habits. That’s not autism. That’s being a person.</p>

<p>Autism is not a personality quirk dialed up to eleven. It’s a fundamentally different way of processing the world—sensory input, social information, language, emotion, all of it. Saying “we’re all a little autistic” because you like your desk organized is like saying “we’re all a little paraplegic” because your legs get tired.</p>

<p>The comment is meant to be sympathetic, I think. It’s an attempt to relate, to find common ground. But what it actually does is minimize. It says: your struggles aren’t real, because everyone has them. It flattens a neurological difference into a relatable quirk. It erases the actual experience of being autistic by pretending everyone shares it.</p>

<p>You’re not a little autistic. You’re just a person who sometimes feels awkward. Those are different things. One of them comes with a lifetime of not fitting in, of exhausting yourself to appear normal, of being misunderstood in ways that are hard to explain to people who’ve never experienced it. The other one is called “being human.”</p>

<p>If you want to relate, just say “that sounds hard” and move on. “What’s really hard for you?” if you really want to brighten an autist’s day. Don’t claim membership in a club you’re not in.</p>

<h2 id="what-autistic-actually-looks-like-in-my-case">What autistic actually looks like, in my case</h2>

<ul>
  <li>Listening to the same three artists on repeat for more than a decade (Purity Ring, Nero, and Flume, if you’re wondering);</li>
  <li>Eating the same breakfast, because it works and change is bad;</li>
  <li>Owning nine of the exact same shirt and NOT because Steve Jobs also wore the same shirt;</li>
  <li>Having to leave social events early because my brain is full;</li>
  <li>Reading a room wrong and not realizing until months later, or sometimes never;</li>
  <li>Scripting conversations in advance so I don’t say something weird, and by in advance I mean for a month;</li>
  <li>Saying something weird anyway always;</li>
  <li>Having to consciously remember to ask people questions about themselves and do so without sounding like a robot, as if it’s natural to not sound like a robot;</li>
  <li>Getting overwhelmed by sounds that other people apparently don’t notice, like the hum of a refrigerator or someone existing near me;</li>
  <li>Struggling to identify what I’m feeling until it becomes a huge problem, including needing to bathroom or eat or drink some water;</li>
  <li>Taking things literally and then being surprised when that wasn’t the point, oh;</li>
  <li>Waiting for trains at railroad crossings on purpose, because trains</li>
</ul>

<p>None of this looks like anything from the outside. You’d just see a guy eating eggs, wearing a slim black shirt, leaving a party early because he’s hot and with some babe, parked at a railroad crossing watching freight cars go by. Normal enough. Quirky at worst.</p>

<h2 id="the-performance">The performance</h2>

<p>Here’s what people miss: I’m not effortlessly normal. I’m <em>performing</em> normal, constantly, and the performance is exhausting.</p>

<p>Every social interaction has a script I’m running in the background. What’s my face doing. Is this the appropriate amount of eye contact? Did I let them finish talking? (Narrator: <em>no</em>.) Should I laugh now, was that a joke or serious? Ope, too late, I already responded wrong.</p>

<p>Neurotypical people—as far as I can tell—don’t have to think about this. It just happens for them. For me, it’s manual. Every time.</p>

<p>Devon Price’s <em><a href="https://bookshop.org/p/books/unmasking-autism-discovering-the-new-faces-of-neurodiversity-devon-price/17363182">Unmasking Autism</a></em> describes this performance as “<a href="https://link.springer.com/article/10.1007/s10803-021-04987-w">masking</a>“—the process of suppressing autistic traits and mimicking neurotypical behavior to fit in. <a href="https://pubmed.ncbi.nlm.nih.gov/36601637/">Most autistic people do it</a>. Many don’t even realize they’re doing it until they burn out.</p>

<p>The “you don’t look autistic” comment is actually evidence <em>of</em> the autism. We fucking did it! I don’t look autistic because I’ve spent decades learning how to not look autistic. That’s the whole thing. You’re not observing the absence of autism. You’re observing the presence of a very expensive coping mechanism.</p>

<h2 id="the-cost-of-masking">The cost of masking</h2>

<p><a href="https://molecularautism.biomedcentral.com/articles/10.1186/s13229-021-00421-1">Masking isn’t free</a>. It’s a loan against your future self, and the interest rate is brutal.</p>

<p>The energy it takes to perform normalcy all day—monitoring your face in real-time as if it’s a fucking websocket, modulating your voice, suppressing stims, forcing eye contact, following social scripts—that energy has to come from somewhere. And it comes from the reserves you’d otherwise use for, you know, living.</p>

<p><a href="https://cambridge.org/core/product/C3EA6C440A3818257C58BC099CD23A39">Autistic burnout</a> is a real thing. It’s not regular burnout, the kind where you’re tired from working too hard. It’s a specific kind of collapse that comes from sustained masking. Your ability to function just degrades like toilet paper. Skills you had disappear. Sensory sensitivities get worse. The mask slips, or cracks, or falls off entirely because you don’t have the energy to hold it up anymore.</p>

<p>There have been many periods where I couldn’t make myself do basic things, including shower or get out of bed. And no, it wasn’t depression. There have been instances where social interaction became impossible, not just difficult; I couldn’t hear what the person was saying and I couldn’t tell them because I couldn’t use words. There have been times where the performance I’d been maintaining for years just stopped being sustainable. It’s not laziness, though it can look like it. It’s not anxiety. It’s the bill coming due for decades of pretending to be someone you’re not.</p>

<p>The worst part is that the people around you don’t understand what’s happening because you don’t even have the words to identify what you’re feeling in the first place. You were fine before! You were high-functioning. What changed? Nothing changed—you just ran out of the energy to keep faking it. But try explaining that to someone who never knew you were faking in the first place.</p>

<h2 id="late-diagnosis">Late diagnosis</h2>

<p>I wasn’t diagnosed until 2022. This is common for people who can mask well enough to get by—especially if you’re not a white boy who fits the Kanner profile.</p>

<p>Getting diagnosed late is a strange experience. On one hand, relief. There’s a name for this. There’s a reason I’ve always felt like I was running different software than everyone else. I’m not broken—or at least, not broken in the way I thought. The struggles make sense now. The failures make sense. The constant low-grade exhaustion of existing in a world that wasn’t built for my brain: it has a name.</p>

<p>On the other hand, grief. How much easier would things have been if I’d known earlier? How many situations did I misread, how many relationships did I fumble, how many opportunities did I miss because I didn’t have the framework to understand what was happening? How much of my life was spent masking without knowing I was masking, burning energy on a performance I didn’t know I was giving?</p>

<p>You start recontextualizing everything. That time you got fired for being “difficult”—maybe you were just direct in a way that neurotypical people read as rude. That relationship that ended because you “didn’t care enough”—maybe you cared a lot and just didn’t express it in the expected ways. That period of burnout you thought was depression—maybe it was autistic burnout and nobody knew, including you.</p>

<p>Reading <em><a href="https://bookshop.org/p/books/unmasking-autism-discovering-the-new-faces-of-neurodiversity-devon-price/17363182">Unmasking Autism</a></em> while already knowing I was autistic felt like someone narrating my life. Reading <em><a href="https://bookshop.org/p/books/a-little-less-broken-on-autism-attachment-and-finding-home-megan-anna-neff/21049566">A Little Less Broken</a></em> felt like permission to stop pretending the hard parts weren’t hard.</p>

<h2 id="the-reveal">The reveal</h2>

<p>Telling someone you’re autistic after they’ve known you for a while produces interesting results.</p>

<p>Some people become instant experts. They’ve read an article, or they watched a documentary once, or their cousin’s kid is autistic, and suddenly they’re pouring out everything they know about autism as if I haven’t been living it for decades. They explain my own condition to me. I nod along. It’s easier than correcting them, and takes less energy, and I’ll look less like an asshole.</p>

<p>Some people start listing every person they think might be autistic. Their coworker who’s “a little weird.” Their uncle who “just doesn’t get social cues.” Their ex who was “probably on the spectrum.” They’re building a mental roster and they want me to know they’ve been paying attention. What they don’t do is offer to introduce me to their hot friend. Apparently being autistic makes me safe to talk to but not safe to set up.</p>

<p>Most people don’t treat me differently at all. Nothing changes. This is fine. Ideal, even. I didn’t tell you so you’d act different. I told you so you’d have context for when I inevitably do something that doesn’t make sense to you.</p>

<p>The best reactions are something like: “Oh, cool. That explains why you’re an asshole sometimes. Not that you’re an asshole.” Acknowledgment without drama. An update to their mental model of me without a whole recalibration. Yeah, sometimes I’m blunt, or I miss something obvious, or I bail on plans because I’m overstimulated. Now you know why. Moving on.</p>

<p>Another great reaction is “let me know what I can do to make your life easier.”</p>

<h2 id="the-spectrum-thing">The spectrum thing</h2>

<p>When people hear “autism spectrum,” they picture a line. On one end: not autistic. On the other end: very autistic. And everyone with autism is somewhere on that line, with “high-functioning” people closer to the normal end and “low-functioning” people closer to the severe end.</p>

<p>This is wrong. That’s not how the spectrum works.</p>

<p>The spectrum isn’t a line from “a little autistic” to “very autistic.” It’s <a href="https://en.wikipedia.org/wiki/History_of_autism">more like a radar chart</a>, or a color wheel, or one of those D&amp;D character stat sheets. There are multiple dimensions—social communication, sensory processing, repetitive behaviors, executive function, emotional regulation, and more—and each one varies independently.</p>

<p>You can be “high” in one area and “low” in another. You can struggle massively with sensory input and have relatively few issues with verbal communication. You can be great at pattern recognition and terrible at reading faces. You can hold down a demanding job and be completely unable to make a phone call.</p>

<p>When someone says they’re “on the spectrum,” they’re not telling you where they fall on a line from normal to weird. They’re telling you they have a particular constellation of traits that varies across multiple axes. Two autistic people can have almost nothing in common in terms of which specific things they struggle with. One might need noise-canceling headphones everywhere; another might not have sensory issues at all but can’t parse sarcasm to save their life.</p>

<p>This is why “you don’t look autistic” is so absurd. There’s no single way to look autistic because there’s no single way to be autistic. The spectrum isn’t a gradient from neurotypical to Rain Man. It’s a multidimensional space, and every autistic person occupies a different point in that space.</p>

<p>The linear interpretation is also where “we’re all a little autistic” comes from. If autism is a line and everyone’s somewhere on it, then sure, maybe you’re just a few notches toward the autistic end. But that’s not the model. The model is: either your brain processes the world in this fundamentally different way, or it doesn’t. And if it does, the specific ways it manifests vary wildly from person to person.</p>

<p>So no, you’re not “a little autistic” because you like routines. You’re just a person who likes routines. I’m autistic because my brain operates on different architecture entirely—and the specifics of that architecture are mine, not a point on a line you can also claim to be on.</p>

<h2 id="high-functioning">“High-functioning”</h2>

<p>I hate this term. Not in a vague political way. I hate it because it’s wrong and it does damage.</p>

<p>“High-functioning” means I can hold a job and pay rent and have conversations without people noticing something is off. From the outside, I function. Great. But here’s what “high-functioning” actually means in practice: your struggles are invisible, so they must not be real.</p>

<p>It means people expect you to be fine all the time, because you’re usually fine. It means when you’re not fine, there’s no framework for it. You were high-functioning yesterday. Why aren’t you high-functioning now? What changed?</p>

<p>Nothing changed. The functioning is inconsistent. It was always inconsistent. You just didn’t see the low-functioning days because I hid or canceled plans or powered through at great personal cost.</p>

<p>The label also implies a binary: high or low. Capable or incapable. But functioning isn’t a single axis. I’m high-functioning at my job. I’m low-functioning at returning phone calls. I’m medium-functioning at keeping friendships alive. I’m completely non-functioning at anything involving the DMV or health insurance or calling to make an appointment.</p>

<p>What looks like high-functioning from the outside is often just well-hidden low-functioning. The performance is good enough that people don’t see the effort. And because they don’t see the effort, they assume there isn’t any. And because they assume there isn’t any, they don’t understand when I hit a wall.</p>

<p>“High-functioning autistic” often just means “autistic person whose struggles are easy to ignore.”</p>

<h2 id="why-this-matters">Why this matters</h2>

<p>I’m not writing this to complain, exactly. Or to educate, though maybe it does that too. I’m writing it because the image of autism in most people’s heads is wrong, and the wrongness has consequences.</p>

<p>When autism only “looks like” the severe, visible version, everyone else gets erased. They don’t get diagnosed. They don’t get support. They spend years or decades thinking something is wrong with them—that they’re broken, or lazy, or just bad at being a person—because nobody told them there was a word for their experience.</p>

<p>When autism only “looks like” intellectual disability, autistic people with normal or high intelligence get dismissed. Their struggles aren’t real because they’re too smart to be autistic. They should be able to just try harder. The mask becomes mandatory because nobody believes them when they take it off.</p>

<p>When “we’re all a little autistic,” the actual autistic experience gets flattened into a quirk. The real challenges disappear into a fog of relatability. Everyone’s a little autistic, so no one is, so what are you even complaining about?</p>

<p>I don’t need people to understand autism completely. I just need them to understand that their mental image is probably incomplete. That the quiet coworker might be masking. That the friend who cancels plans might be burnt out, not flaky. That “you don’t look autistic” isn’t the compliment they think it is.</p>

<h2 id="youre-right-actually">You’re right, actually</h2>

<p>Fine. I don’t look autistic. But also: maybe you don’t know what autistic looks like.</p>

<p>It looks like your coworker who always eats alone and you assume just prefers it that way. It looks like your friend who cancels plans last minute and you think is flaky. It looks like the person who interrupts a lot and you find annoying. It looks like the guy who takes jokes seriously and you think has no sense of humor. It looks like the woman who doesn’t notice you’re upset until you explicitly say so.</p>

<p>It looks like a lot of people you’ve already met and already judged, because they didn’t match the image in your head.</p>

<p>I don’t look autistic. Neither do most autistic people. That’s sort of the point.</p>
]]></content>
    
    <summary>I get this a lot. “You don’t look autistic.” It’s meant as a compliment, I think. Like I’ve done a good job of hiding it. Gold star for passing. Or it’s a remark that implies I don’t have autism because I don’t visibly struggle.
</summary>
    
    
    <category term="personal"/>
    
  </entry>
  
  <entry>
    <title>Going to federal prison for piracy</title>
    <link href="/posts/going-to-federal-prison-for-piracy/" rel="alternate" type="text/html"/>
    <id>/posts/going-to-federal-prison-for-piracy/</id>
    <published>2025-09-18T00:00:00+00:00</published>
    <updated>2025-09-18T00:00:00+00:00</updated>
    <content type="html"><![CDATA[<p>Here’s something I’ve been putting off writing: I spent eighteen months in federal prison. The charge was computer fraud. The context is weirder.</p>

<p>In 2016, I built a website called HeheStreams. It started as a joke—a proof of concept I posted on Reddit after BallStreams, my beloved NBA streaming site, vanished overnight. Nobody picked it up, so I kept working on it. Eventually I slapped a paywall on it, thinking it was ludicrous that anyone would pay for such a thing.</p>

<p>People paid for such a thing.</p>

<h2 id="what-hehestreams-actually-was">What HeheStreams actually was</h2>

<p>HeheStreams let users watch live sports from MLB, NBA, NFL, and NHL. But it wasn’t a typical piracy operation. I wasn’t capturing broadcasts and rehosting them. Users connected to the leagues’ own streams—their infrastructure, their CDNs. The streams were as reliable as the official ones because they were the official ones.</p>

<p>It was, in retrospect, a better product than what the leagues were selling. Users asked if it was legal. I had to explain that no, no sports league licenses content to a website called HeheStreams.</p>

<p>It was an open secret in certain circles. Many of my fellow developers at previous jobs knew about HeheStreams. One happened to be a subscriber before we worked together. That’s the kind of product-market fit you don’t expect from a janky piracy side project.</p>

<p>The site ran from 2016 to 2021.</p>

<p>It wasn’t about the money. It never was.</p>

<p>I made the site for me so I could watch sports. I just had some other people watching with me and didn’t know how to say no.</p>

<p>When I eventually put up a paywall, it was “pay what you think it’s worth.” The first purchase was for $100. I’d hoped nobody would pay—I wanted a social life and didn’t want to be beholden to grown men telling me their janky streaming site was broken. But people kept paying. So I kept building.</p>

<p>And then I felt an obligation to fix the site when grown men told me my shitty basketball streaming website wasn’t working for them. That’s a weird kind of pressure. It’s also, apparently, the seed of product-market fit.</p>

<p>The big draw was bypassing geoblocking that you couldn’t otherwise buy your way out of legally. If you lived in the wrong city for your team, the official services were useless. HeheStreams wasn’t.</p>

<p>I said no to a lot of requests. UFC/MMA/boxing I turned down because their business models revolve around PPV—I wasn’t going to undercut something that depended on individual event revenue. NCAA sports I refused outright. NIL didn’t exist at the time, and I wasn’t going to profit off children. Feature parity mattered to me too. If I couldn’t make it as good as the other sports’ implementations, I didn’t add it.</p>

<p>Not every customer is a good customer. I learned that quickly. The quirks and features of the site filtered for a certain type of user—people who could figure things out, who understood that things would occasionally break, and who didn’t expect me to be their personal concierge.</p>

<p>I treated every message—even transactional emails—as an opportunity to build trust. My copywriting was informal and self-deprecating. I swore at myself when apologizing for things not working. People told me they looked forward to my automated emails. That was a good litmus test.</p>

<p>But scaling personality is nearly impossible. You can’t build a playbook for friendliness. People have bad days they drag into work. I’m guilty of this too. The week after my mom died, I was terse. I have uncomfortable memories of being short with users and not living up to my own standards. I went so far as to tell one user my situation, and he told me that because I’m providing a service, I have to do better.</p>

<p>He churned.</p>

<p>That part of the story deserves its own post.</p>

<h2 id="how-it-ended">How it ended</h2>

<p>In July 2021, the Alliance for Creativity and Entertainment came knocking. Civil settlement. Site goes dark.</p>

<p>Then in October 2021, the U.S. Government decided they weren’t done. I was charged with computer fraud, wire fraud, illicit digital transmission, and—here’s the fun one—extortion.</p>

<p>The extortion charge came from a series of emails with Major League Baseball. I’d found some security vulnerabilities in their systems. Nothing related to streaming—garden-variety web issues. I disclosed them responsibly and wanted to blog about them.</p>

<p>Someone at MLB asked what I “valued” the bugs at. Being me—autistic, methodical—I referenced Shopify’s bug bounty calculator. It spat out something like $150k per bug. I immediately said that was ridiculous since I’d spent maybe ten minutes finding them.</p>

<p>Nuance doesn’t survive email threads.</p>

<p>The autism meant I missed every bit of corporate subtext. That’s not an excuse—just a debugging note about the operating systems involved.</p>

<h2 id="the-plea">The plea</h2>

<p>I signed a plea deal. I pleaded guilty to one count of computer fraud—accessing a protected computer without authorization. The streaming part, I did. No footnotes, no irony.</p>

<p>I was sentenced to three years in federal prison, three years of supervised release, $500,000 in forfeiture, and roughly $3 million in restitution.</p>

<p>My name ended up on ESPN for something that started as a weekend project. That was weird.</p>

<h2 id="the-facility">The facility</h2>

<p>I did my time at FCI Thomson in northwestern Illinois.</p>

<p>If that name sounds familiar, you might know it as “Gitmo North.” The Obama administration originally planned to house Guantanamo Bay detainees there. Congress blocked that, so the Bureau of Prisons bought it instead and turned it into a Special Management Unit—essentially a second supermax, modeled after ADX Florence.</p>

<p>For years, Thomson was one of the deadliest federal prisons in the country. Multiple inmate murders—by staff. The Marshall Project called it “one of the deadliest” facilities in the federal system. In 2023, after a series of homicides and a damning investigation, the BOP finally converted it to a low-security facility. They already paid for it, so they had to make use of it.</p>

<p>The average age of an inmate in the federal system will soon be 55. Many of those inmates will transition to low-security facilities because of how the scoring system works. There are already overcrowding issues and staffing issues—nobody wants to live in the middle of a cornfield.</p>

<h2 id="18-months-inside">18 months inside</h2>

<p>Federal prison is optimized—systemically and socially—for drug cases and sex offenses. There were many instances where there was no “other” checkbox. This had a trickle-down effect: inmates assumed a short, well-groomed male was a sex offender because he didn’t fit the profile.</p>

<p>I had to answer the question “what are you in for?” a lot. People would come back to me saying, “hey, so-and-so was asking what you were in for, lol.” The rumor mill runs on assumptions, and the assumptions aren’t kind to anyone who doesn’t fit the standard profiles.</p>

<p>Sex offenders were treated as significantly less than human. The unwritten rules were clear: don’t associate, don’t share, don’t acknowledge. And I get it—some of those crimes are genuinely heinous. But the blanket dehumanization extended to everyone in that category, regardless of circumstances. There’s no nuance. No consideration for the 19-year-old who slept with his 17-year-old girlfriend. No distinction between the predator and the edge case. There were significantly more of the latter.</p>

<p>I consciously went against those unwritten rules when they didn’t align with my values. Not as some moral stand—I just couldn’t square the blanket cruelty with my own sense of how people should be treated.</p>

<p>For people with longer sentences, prison is their way of life. I was a tourist on the shittiest vacation.</p>

<h2 id="being-autistic-in-federal-prison">Being autistic in federal prison</h2>

<p>I was diagnosed autistic in my 30s, before I went in. Reading <em>Unmasking Autism</em> while incarcerated hit different—suddenly large portions of my life started making sense, and I was processing that while surrounded by people I couldn’t fully read.</p>

<p>I don’t have many sensory sensitivities. Noise is bad, though. Federal prison is loud. Doors. Arguments. The ambient chaos of people living on top of each other. It wears on you. People always moving—always moving—isn’t very fun either.</p>

<p>The bigger issue was social dynamics. Autistic people are notoriously bad at understanding and parsing unwritten rules, and prison runs on unwritten rules. Who you can sit with. What you can say to whom. The subtle signals that mean you’re about to have a problem. I missed things. I misread things. That’s dangerous in an environment where misreading someone can escalate fast.</p>

<p>I adapted better than I expected. I learned I’m more socially capable than I gave myself credit for—better at blending in, better at adjusting. But I was always operating with a slight delay, always parsing things consciously that other people seemed to catch automatically.</p>

<h2 id="reading">Reading</h2>

<p>Here’s something I didn’t expect: I read books. A lot of them. Before prison, I’d never finished one—not even in high school.</p>

<p>My favorites were <em>Determined</em>, <em>Behave</em>, <em>Salt Fat Acid Heat</em>, <em>Unmasking Autism</em>, <em>Enigma</em>, and <em>Neurotribes</em>. Heavy on the neuroscience and autism literature. Tracks.</p>

<h2 id="what-i-learned-about-myself">What I learned about myself</h2>

<p>Prison was weird for my own growth cycle. I didn’t become a different person—just one who understands himself better. Not in a memoir-worthy way.</p>

<p>I’m more tolerant of different people and a lot more worried about incarcerated individuals than I was before. Those inside aren’t set up for success in the free world. I’m an outlier, and I know that. The problem-solver in me has no idea how to fix it, or if it can be fixed within our current system.</p>

<p>Inmates—regardless of crime—need real opportunities while incarcerated that actually prepare them for life afterward. Right now, we fail at that completely.</p>

<h2 id="getting-out">Getting out</h2>

<p>I was released in August of 2025. Finding work was surprisingly effortless, considering the job market. I had six offers within a few weeks.</p>

<p>I took the one where the founder showed up in a sports jersey to our video chat, knowing exactly what I went to prison for.</p>

<p>He’s since told the team that one of the factors in hiring me was the success of the site itself—the product instincts, the technical execution, the fact that I’d built something people wanted badly enough to pay for even when they knew it was illegal. That acknowledgment meant a lot. It was a signal that the thing I’d done wrong was also, in some way, evidence of something I could do right.</p>

<p>It’s been a good fit.</p>

<h2 id="the-same-instinct-pointed-differently">The same instinct, pointed differently</h2>

<p>I don’t think of it as “life after prison.” It’s just life. Same problems, different points of view. The same ingredients rearranged. I still get up, write code, and think about building all the time. There’s a latte in my routine now. That’s different I guess.</p>

<p>Another difference is that I’m more deliberate about the things I say yes to. I try to build stuff that doesn’t just work, but feels aligned with something bigger than my own curiosity. You start caring about that after things go sideways once.</p>

<p>Curiosity is neutral. It’s what you point it at that matters. That’s both empowering and terrifying—especially when your curiosity once built something that landed you federal charges. My curiousity doesn’t extend itself to other crime-y things, thankfully.</p>

<p>I haven’t lost the impulse that got me here. The same instinct that made me want to take apart systems and rebuild them is the one that drove HeheStreams, and it’s the same one that makes me good at my job now.</p>

<h2 id="on-vulnerability-disclosure">On vulnerability disclosure</h2>

<p>Every company should have a standardized playbook for handling security disclosures. Every single one. Not a “we’ll get back to you” email. Not radio silence. Not calling the FBI.</p>

<p>A clear, published process. A timeline. A point of contact. Protections for the reporter.</p>

<p>This exists for whistleblowers. It should exist for bug reporters too.</p>

<p>One thing that bothers me is that corporations engage in behavior that’s ethically far worse—price-fixing, wage theft, environmental destruction—and pay fines that amount to rounding errors on their quarterly earnings.</p>

<p>I’m not saying I didn’t break the law. I totally and absolutely did. Asymmetry is hard to ignore.</p>

<h2 id="on-the-extortion-narrative">On the “extortion” narrative</h2>

<p>This was paraded and I’m still unhappy about it.</p>

<h2 id="why-im-writing-this">Why I’m writing this</h2>

<p>There’s a certain freedom in owning your story publicly. People can’t weaponize what you’ve already made peace with.</p>

<p>The explainer website originally existed privately, and I’d pass it along with my resume—which included my mention of prison—so that there’s context. It was never meant for a wide audience.</p>

<p>It hit the front page of HackerNews twice (both times posted by another user) lingered for a bit. There’s some validation there—not validation for <em>what</em> I did but the context wasn’t as poorly received as I had anticipated, had the public ever had a chance to see it. When you’re incarcerated, there’s some weird voodoo that convinces you that your crime is like everyone else’s. I was incarcerated at the same facility, yes, but I’d argue that second-degree murder isn’t the same as my charge. The system shapes you to think that way. It did me, at least. I had that mentality getting out. The little ego boost I got from HackerNews jas been helpful in the healing sense.</p>

<p>Now I’m out and I can have a little bit of coming-out party.</p>

<p>I recently did the same thing at work—gave a slideshow during an all-hands that started as a standard “about me” and then pivoted to “oh yeah, also.” Being able to shape the narrative and tell my side before someone Googles me and finds the slanted reporting has proven helpful. I told them the truth is usually somewhere between what the DOJ says and what actually happened.</p>

<p>I’m still the same person. Just with better self-knowledge, I guess.</p>

<hr />

<p><em>For more detail on the charges, the site, or the rest of it: <a href="https://prison.josh.mn">prison.josh.mn</a></em></p>
]]></content>
    
    <summary>Here’s something I’ve been putting off writing: I spent eighteen months in federal prison. The charge was computer fraud. The context is weirder.
</summary>
    
    
    <category term="personal"/>
    
  </entry>
  
  <entry>
    <title>HTTP 451</title>
    <link href="/451/" rel="alternate" type="text/html"/>
    <id>/451/</id>
    <published>2024-12-31T00:00:00+00:00</published>
    <updated>2024-12-31T00:00:00+00:00</updated>
    <content type="html"><![CDATA[<div class="document-header">
  <div class="document-stamp">SEALED</div>
  <div class="document-case">Case No. 22-cr-00350 (S.D.N.Y.)</div>
</div>

<div class="error-code">451</div>

<div class="error-title">Unavailable For Legal Reasons</div>

<div class="error-desc">
The content you are attempting to access has been restricted due to a legal demand.
</div>

<div class="prison-letter">
  <div class="letter-header">
    <span>FCI Thomson</span>
    <span>Inspected</span>
  </div>
  <div class="letter-body">
    <p>Hey.</p>

    <p>So I'm in federal prison now. Long story. Actually it's not that long, it's just ██████████. Shit happens.</p>

    <p>Anyway.</p>

    <p>I've been reading a lot. The library here has exactly one book I care about: <em>HTML for the World Wide Web</em> from 1999. There's a chapter on "publishing your web page to GeoCities." I've read it four times. Not because it's useful. Because it's the closest thing to a religious experience I can have in here.</p>

    <p>My cellie is a guy named Big Ron. He's in for ██████████ ███ ████████. I asked what he did before prison. He said "logistics." I said "oh like supply chain?" He just stared at me. We don't talk about his work anymore.</p>

    <p>Big Ron asked what I did. I tried to explain. He said "so like hacking?" I said "not exactly." He said "can you hack my baby mama's Instagram." I said no. He asked every day for three weeks. I finally told him my buddy on the outside tried but fucked it up and now it's "too encrypted." He respects me now.</p>

    <p>The food here follows a strict schedule. Tuesday is chicken. Wednesday is burger. Thursday is chicken. Friday is fish. Saturday and Sunday vary, which means chicken. Monday is ██████████ but it looks like chicken. I'm autistic. I like routine. But holy shit, not like this. This is too much routine. I now understand why people fight over commissary ramen.</p>

    <p>Speaking of the economy. Stamps are 50 cents each but a book of them costs $10. A $2.10 bag of chips sells for $8 if you want it now. Everything else is double. Someone tried to explain the logic to me using a napkin diagram and I swear to god it was more sophisticated than most DeFi whitepapers I've ever fucking read. The invisible hand works different in here.</p>

    <p>I started teaching a business class. Unofficial. Just me and whoever shows up. We talk about pricing, margins, customer acquisition, market positioning. The drug dealers are my best students by far. They already understand all of it intuitively. I'm basically just giving them vocabulary. One guy named Terrance took notes and said "this is just what I was doing but with words." He's in for ██████████. Dude had a 40% margin before he got caught. That's better than most SaaS companies.</p>

    <p>The K2 problem here is fucking wild. Guys smoke that synthetic shit and just ████████ ███ ██████████. Last week someone thought he was a ██████ and tried to ████████ ███ █████████. Another guy just stood in the corner for six hours making a noise. Not words. Just a noise. The COs didn't even react. That's how often this shit happens.</p>

    <p>I've joined the book club. It's me, a guy who exclusively reads Tom Clancy, a guy who exclusively reads the Bible, and a guy named Professor who claims he wrote the original draft of ████████████ before ██████ ██████ stole it. We're currently reading <em>The Count of Monte Cristo</em>. Every week Professor explains that Dantes' revenge was "amateur hour" compared to what he would have done. Nobody asks follow up questions.</p>

    <p>I've been playing a lot of basketball. Not real basketball. I can't do that. But pig, horse, around the world—that I can do. Turns out I'm actually good at it. And whenever I beat a black dude, which happens more than you'd think, all the other black dudes give him absolute hell for the rest of the day. "You lost to the white boy?" "The hacker got you?" It's relentless. I have basketball street cred now. This is the strangest timeline.</p>

    <p>Also whenever the TVs fuck up everyone looks at me. "Yo, hacker, fix the TV." I'm not that kind of hacker. That's not how any of this works. But I go up there and unplug it and plug it back in and it works and now I'm the TV guy too. The bar is so low it's underground.</p>

    <p><strong>Things I've learned:</strong></p>
    <ul>
      <li>I am bad at spades. Catastrophically bad. People have lost commissary betting on me. Not with me. On me. Against their own goddamn interests. That's how bad.</li>
      <li>The optimal shower time is ████ because ██████████.</li>
      <li>If someone asks if you're "good," the correct answer is always yes. Even if you're not. Especially if you're not.</li>
      <li>Honey buns do not expire. Or if they do, nobody here cares.</li>
    </ul>

    <p><strong>Things I miss:</strong></p>
    <ul>
      <li>████████</li>
      <li>██████████ ███ ██████</li>
      <li>Mechanical keyboards</li>
      <li>Choosing what to eat</li>
      <li>Making ██████████ amounts of █████</li>
    </ul>

    <p><strong>Things I don't miss:</strong></p>
    <ul>
      <li>Twitter</li>
      <li>Linkedin</li>
      <li>People asking me to look at their startup idea</li>
      <li>People asking me what ████████████ is worth</li>
      <li>Standing desk discourse</li>
    </ul>

    <p>Someone started a rumor that I ██████████ ███████ ████████ ██████ from ██████. I didn't correct them. The social capital has been useful as hell. A guy gave me his chicken quarter last week just because he "respects the hustle." I don't know what hustle he thinks I did. I took the chicken.</p>

    <p>My lawyer says ████████ ██████ ███ ████████████ ██████ which is ██████████ but also ████████. So that's where we are.</p>

    <p>I've had a lot of time to think. About life. About choices. About whether I'd do anything differently. The answer is ██████████ ███ ██████████ ███ ████ ██████████ ████████ ██████ ███ ████████ ██████████. But that's between me and my journal.</p>

    <p>The sunsets here are actually beautiful. That feels weird to say. But they are. There's a window in the common area and every evening the sky does this thing where it goes all orange and purple and for like ten minutes everyone just shuts up and watches. Even the guys who ██████████ ███ ████████. It's the only time this place is quiet. Well, except when someone's on K2. Then it's never quiet.</p>

    <p>Anyway. I'm fine. Really. The food is ██████, the company is ██████████, and my lawyer says I'll be out in ██████████. So basically living the dream.</p>

    <p>Write back if you want. Or don't. Mail call is the highlight of my day either way, and I mean that in the most depressing way possible.</p>

    <p>Later.</p>

    <p>P.S. If anyone asks, I ██████████ ███ ████████████ ██████ ███ ██████████. That's my official position now.</p>

  </div>
</div>

<div class="error-citation">
USA v. Streit et al., 22-cr-00350 (S.D.N.Y.)<br />
18 U.S.C. § 1030(a)(2)<br /><br />
The HTTP 451 status code is a reference to Ray Bradbury's novel <em>Fahrenheit 451</em>, in which books are outlawed.
</div>
]]></content>
    
    <summary>
  SEALED
  Case No. 22-cr-00350 (S.D.N.Y.)

</summary>
    
    
    <category term="personal"/>
    
  </entry>
  
  <entry>
    <title>Gem configuration patterns</title>
    <link href="/posts/gem-configuration-patterns/" rel="alternate" type="text/html"/>
    <id>/posts/gem-configuration-patterns/</id>
    <published>2023-08-12T00:00:00+00:00</published>
    <updated>2023-08-12T00:00:00+00:00</updated>
    <content type="html"><![CDATA[<p>Configuration is often the first thing users interact with in your gem. It’s also the part most gem authors spend the least time thinking about—which explains a lot. This post covers the patterns you’ll encounter and implement, ordered from the simplest to the most involved.</p>

<h2 id="module-accessors">Module accessors</h2>

<p>The simplest possible approach. You expose a few attributes on your gem’s main module and call it a day.</p>

<div class="playground" data-runtime="ruby">
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">module</span> <span class="nn">MyGem</span>
  <span class="k">class</span> <span class="o">&lt;&lt;</span> <span class="nb">self</span>
    <span class="nb">attr_accessor</span> <span class="ss">:api_key</span><span class="p">,</span> <span class="ss">:timeout</span><span class="p">,</span> <span class="ss">:debug</span>
  <span class="k">end</span>

  <span class="nb">self</span><span class="p">.</span><span class="nf">timeout</span> <span class="o">=</span> <span class="mi">30</span>
  <span class="nb">self</span><span class="p">.</span><span class="nf">debug</span> <span class="o">=</span> <span class="kp">false</span>
<span class="k">end</span>

<span class="no">MyGem</span><span class="p">.</span><span class="nf">api_key</span> <span class="o">=</span> <span class="s2">"sk-123"</span>
<span class="no">MyGem</span><span class="p">.</span><span class="nf">timeout</span> <span class="o">=</span> <span class="mi">60</span>

<span class="nb">puts</span> <span class="s2">"api_key: </span><span class="si">#{</span><span class="no">MyGem</span><span class="p">.</span><span class="nf">api_key</span><span class="si">}</span><span class="s2">"</span>
<span class="nb">puts</span> <span class="s2">"timeout: </span><span class="si">#{</span><span class="no">MyGem</span><span class="p">.</span><span class="nf">timeout</span><span class="si">}</span><span class="s2">"</span>
<span class="nb">puts</span> <span class="s2">"debug: </span><span class="si">#{</span><span class="no">MyGem</span><span class="p">.</span><span class="nf">debug</span><span class="si">}</span><span class="s2">"</span></code></pre></div></div>
<div class="playground-controls">
<div class="playground-controls-buttons">
<button class="playground-run" disabled="">Run</button>
<button class="playground-reset" disabled="">Reset</button>
</div>
<span class="playground-controls-note">loads ~35MB Ruby environment</span>
</div>
<div class="playground-output"><pre><code>api_key: sk-123
timeout: 60
debug: false</code></pre></div>
</div>

<p>This works for gems with one or two options. It falls apart quickly once you need defaults that depend on each other, validation, or any structure. Most gems outgrow this within a few releases—usually right around the time someone opens an issue asking why their configuration disappeared between requests.</p>

<h2 id="block-based-configuration">Block-based configuration</h2>

<p>The standard pattern. Users call a <code>configure</code> method with a block, and you yield a configuration object.</p>

<div class="playground" data-runtime="ruby">
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">module</span> <span class="nn">MyGem</span>
  <span class="k">class</span> <span class="nc">Configuration</span>
    <span class="nb">attr_accessor</span> <span class="ss">:api_key</span><span class="p">,</span> <span class="ss">:timeout</span><span class="p">,</span> <span class="ss">:debug</span>

    <span class="k">def</span> <span class="nf">initialize</span>
      <span class="vi">@timeout</span> <span class="o">=</span> <span class="mi">30</span>
      <span class="vi">@debug</span> <span class="o">=</span> <span class="kp">false</span>
    <span class="k">end</span>
  <span class="k">end</span>

  <span class="k">class</span> <span class="o">&lt;&lt;</span> <span class="nb">self</span>
    <span class="k">def</span> <span class="nf">configuration</span>
      <span class="vi">@configuration</span> <span class="o">||=</span> <span class="no">Configuration</span><span class="p">.</span><span class="nf">new</span>
    <span class="k">end</span>

    <span class="k">def</span> <span class="nf">configure</span>
      <span class="k">yield</span><span class="p">(</span><span class="n">configuration</span><span class="p">)</span>
    <span class="k">end</span>
  <span class="k">end</span>
<span class="k">end</span>

<span class="no">MyGem</span><span class="p">.</span><span class="nf">configure</span> <span class="k">do</span> <span class="o">|</span><span class="n">config</span><span class="o">|</span>
  <span class="n">config</span><span class="p">.</span><span class="nf">api_key</span> <span class="o">=</span> <span class="s2">"sk-123"</span>
  <span class="n">config</span><span class="p">.</span><span class="nf">timeout</span> <span class="o">=</span> <span class="mi">60</span>
<span class="k">end</span>

<span class="nb">puts</span> <span class="s2">"api_key: </span><span class="si">#{</span><span class="no">MyGem</span><span class="p">.</span><span class="nf">configuration</span><span class="p">.</span><span class="nf">api_key</span><span class="si">}</span><span class="s2">"</span>
<span class="nb">puts</span> <span class="s2">"timeout: </span><span class="si">#{</span><span class="no">MyGem</span><span class="p">.</span><span class="nf">configuration</span><span class="p">.</span><span class="nf">timeout</span><span class="si">}</span><span class="s2">"</span></code></pre></div></div>
<div class="playground-controls">
<div class="playground-controls-buttons">
<button class="playground-run" disabled="">Run</button>
<button class="playground-reset" disabled="">Reset</button>
</div>
<span class="playground-controls-note">loads ~35MB Ruby environment</span>
</div>
<div class="playground-output"><pre><code>api_key: sk-123
timeout: 60</code></pre></div>
</div>

<p>This is what most gems use, and for good reason. It scales reasonably well, keeps defaults in one place, and the syntax is familiar enough that users can configure your gem without reading the docs. Whether they will read the docs is a separate problem.</p>

<h2 id="configuration-with-validation">Configuration with validation</h2>

<p>Once you have a configuration class, you can add validation. Catch mistakes early rather than letting them surface as cryptic errors deep in your code.</p>

<div class="playground" data-runtime="ruby">
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">module</span> <span class="nn">MyGem</span>
  <span class="k">class</span> <span class="nc">Configuration</span>
    <span class="nb">attr_reader</span> <span class="ss">:api_key</span><span class="p">,</span> <span class="ss">:timeout</span><span class="p">,</span> <span class="ss">:retry_count</span>

    <span class="k">def</span> <span class="nf">initialize</span>
      <span class="vi">@timeout</span> <span class="o">=</span> <span class="mi">30</span>
      <span class="vi">@retry_count</span> <span class="o">=</span> <span class="mi">3</span>
    <span class="k">end</span>

    <span class="k">def</span> <span class="nf">api_key</span><span class="o">=</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
      <span class="k">raise</span> <span class="no">ArgumentError</span><span class="p">,</span> <span class="s2">"api_key cannot be blank"</span> <span class="k">if</span> <span class="n">value</span><span class="p">.</span><span class="nf">nil?</span> <span class="o">||</span> <span class="n">value</span><span class="p">.</span><span class="nf">empty?</span>
      <span class="vi">@api_key</span> <span class="o">=</span> <span class="n">value</span>
    <span class="k">end</span>

    <span class="k">def</span> <span class="nf">timeout</span><span class="o">=</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
      <span class="k">raise</span> <span class="no">ArgumentError</span><span class="p">,</span> <span class="s2">"timeout must be positive"</span> <span class="k">unless</span> <span class="n">value</span><span class="p">.</span><span class="nf">is_a?</span><span class="p">(</span><span class="no">Numeric</span><span class="p">)</span> <span class="o">&amp;&amp;</span> <span class="n">value</span> <span class="o">&gt;</span> <span class="mi">0</span>
      <span class="vi">@timeout</span> <span class="o">=</span> <span class="n">value</span>
    <span class="k">end</span>

    <span class="k">def</span> <span class="nf">retry_count</span><span class="o">=</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
      <span class="k">raise</span> <span class="no">ArgumentError</span><span class="p">,</span> <span class="s2">"retry_count must be between 0 and 10"</span> <span class="k">unless</span> <span class="p">(</span><span class="mi">0</span><span class="o">..</span><span class="mi">10</span><span class="p">).</span><span class="nf">cover?</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
      <span class="vi">@retry_count</span> <span class="o">=</span> <span class="n">value</span>
    <span class="k">end</span>
  <span class="k">end</span>
<span class="k">end</span>

<span class="n">config</span> <span class="o">=</span> <span class="no">MyGem</span><span class="o">::</span><span class="no">Configuration</span><span class="p">.</span><span class="nf">new</span>
<span class="n">config</span><span class="p">.</span><span class="nf">api_key</span> <span class="o">=</span> <span class="s2">"sk-123"</span>
<span class="n">config</span><span class="p">.</span><span class="nf">timeout</span> <span class="o">=</span> <span class="mi">60</span>
<span class="nb">puts</span> <span class="s2">"valid config: api_key=</span><span class="si">#{</span><span class="n">config</span><span class="p">.</span><span class="nf">api_key</span><span class="si">}</span><span class="s2">, timeout=</span><span class="si">#{</span><span class="n">config</span><span class="p">.</span><span class="nf">timeout</span><span class="si">}</span><span class="s2">"</span>

<span class="k">begin</span>
  <span class="n">config</span><span class="p">.</span><span class="nf">timeout</span> <span class="o">=</span> <span class="o">-</span><span class="mi">5</span>
<span class="k">rescue</span> <span class="no">ArgumentError</span> <span class="o">=&gt;</span> <span class="n">e</span>
  <span class="nb">puts</span> <span class="s2">"caught: </span><span class="si">#{</span><span class="n">e</span><span class="p">.</span><span class="nf">message</span><span class="si">}</span><span class="s2">"</span>
<span class="k">end</span></code></pre></div></div>
<div class="playground-controls">
<div class="playground-controls-buttons">
<button class="playground-run" disabled="">Run</button>
<button class="playground-reset" disabled="">Reset</button>
</div>
<span class="playground-controls-note">loads ~35MB Ruby environment</span>
</div>
<div class="playground-output"><pre><code>valid config: api_key=sk-123, timeout=60
caught: timeout must be positive</code></pre></div>
</div>

<p>The error messages should be specific. “Invalid timeout” tells users nothing; “timeout must be positive” tells them exactly what went wrong. Your future self debugging a production issue at 2am will thank you.</p>

<h2 id="nested-configuration">Nested configuration</h2>

<p>When your gem has distinct subsystems, nested configuration keeps things organized. Users access settings via <code>config.api.timeout</code> rather than <code>config.api_timeout</code>.</p>

<pre><code class="language-ruby">module MyGem
  class Configuration
    attr_reader :api, :cache, :logging

    def initialize
      @api = ApiConfig.new
      @cache = CacheConfig.new
      @logging = LoggingConfig.new
    end
  end

  class ApiConfig
    attr_accessor :endpoint, :timeout, :retries

    def initialize
      @endpoint = "https://api.example.com"
      @timeout = 30
      @retries = 3
    end
  end

  class CacheConfig
    attr_accessor :enabled, :ttl, :store

    def initialize
      @enabled = true
      @ttl = 3600
      @store = :memory
    end
  end

  class LoggingConfig
    attr_accessor :level, :logger

    def initialize
      @level = :info
      @logger = nil
    end
  end
end
</code></pre>

<p>Users configure it like this:</p>

<pre><code class="language-ruby">MyGem.configure do |config|
  config.api.timeout = 60
  config.api.retries = 5
  config.cache.enabled = false
  config.logging.level = :debug
end
</code></pre>

<p>This pattern makes sense when you have clear boundaries between concerns. If your gem only has five or six settings total, nesting just gives users more dots to type for no reason.</p>

<h2 id="per-instance-vs-global-configuration">Per-instance vs global configuration</h2>

<p>Some gems need both global defaults and per-instance overrides. A Stripe-style client that can operate with different API keys is the canonical example.</p>

<pre><code class="language-ruby">module MyGem
  class &lt;&lt; self
    def configuration
      @configuration ||= Configuration.new
    end

    def configure
      yield(configuration)
    end

    # convenience method using global config
    def client
      @client ||= Client.new(configuration)
    end
  end

  class Configuration
    attr_accessor :api_key, :timeout

    def initialize(api_key: nil, timeout: 30)
      @api_key = api_key
      @timeout = timeout
    end

    def dup
      Configuration.new(api_key: api_key, timeout: timeout)
    end
  end

  class Client
    attr_reader :config

    def initialize(config = nil)
      @config = config || MyGem.configuration.dup
    end

    def request(path)
      # uses @config.api_key, @config.timeout
    end
  end
end
</code></pre>

<p>Users can use the global client or create their own:</p>

<pre><code class="language-ruby"># global configuration
MyGem.configure do |config|
  config.api_key = "sk-default"
end

# use global client
MyGem.client.request("/users")

# per-instance client with different credentials
other_client = MyGem::Client.new(
  MyGem::Configuration.new(api_key: "sk-other")
)
other_client.request("/users")
</code></pre>

<p>The <code>dup</code> method matters here. Without it, per-instance clients that start from the global config would share the same configuration object, and changes to one would affect all of them. I have debugged this exact issue more times than I care to admit.</p>

<h2 id="rails-integration">Rails integration</h2>

<p>If your gem integrates with Rails, you probably want an initializer generator and possibly a Railtie.</p>

<p>The generator creates a config file:</p>

<pre><code class="language-ruby"># lib/generators/my_gem/install_generator.rb
module MyGem
  module Generators
    class InstallGenerator &lt; Rails::Generators::Base
      source_root File.expand_path("templates", __dir__)

      def copy_initializer
        template "initializer.rb", "config/initializers/my_gem.rb"
      end
    end
  end
end
</code></pre>

<pre><code class="language-ruby"># lib/generators/my_gem/templates/initializer.rb
MyGem.configure do |config|
  config.api_key = ENV["MY_GEM_API_KEY"]
  config.timeout = 30
end
</code></pre>

<p>Users run <code>rails generate my_gem:install</code> and get a starting point. They also get the satisfaction of feeling like they configured something, even if they just accepted all the defaults.</p>

<p>A Railtie lets you hook into Rails’ lifecycle:</p>

<pre><code class="language-ruby"># lib/my_gem/railtie.rb
module MyGem
  class Railtie &lt; Rails::Railtie
    initializer "my_gem.configure" do
      MyGem.configuration.logger ||= Rails.logger
    end

    config.after_initialize do
      if MyGem.configuration.api_key.nil?
        Rails.logger.warn "[MyGem] No API key configured"
      end
    end
  end
end
</code></pre>

<p>The Railtie is autoloaded when Rails loads your gem—as long as you require it in your main gem file when Rails is present:</p>

<pre><code class="language-ruby"># lib/my_gem.rb
require "my_gem/railtie" if defined?(Rails::Railtie)
</code></pre>

<h2 id="dsl-based-configuration">DSL-based configuration</h2>

<p>When block-based configuration isn’t expressive enough, a DSL can help. This is common in gems that configure complex structures like routes, state machines, or pipelines.</p>

<div class="playground" data-runtime="ruby">
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">module</span> <span class="nn">MyGem</span>
  <span class="k">class</span> <span class="nc">Configuration</span>
    <span class="nb">attr_reader</span> <span class="ss">:endpoints</span>

    <span class="k">def</span> <span class="nf">initialize</span>
      <span class="vi">@endpoints</span> <span class="o">=</span> <span class="p">{}</span>
    <span class="k">end</span>

    <span class="k">def</span> <span class="nf">endpoint</span><span class="p">(</span><span class="nb">name</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">block</span><span class="p">)</span>
      <span class="n">ep</span> <span class="o">=</span> <span class="no">EndpointConfig</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="nb">name</span><span class="p">)</span>
      <span class="n">ep</span><span class="p">.</span><span class="nf">instance_eval</span><span class="p">(</span><span class="o">&amp;</span><span class="n">block</span><span class="p">)</span>
      <span class="vi">@endpoints</span><span class="p">[</span><span class="nb">name</span><span class="p">]</span> <span class="o">=</span> <span class="n">ep</span>
    <span class="k">end</span>
  <span class="k">end</span>

  <span class="k">class</span> <span class="nc">EndpointConfig</span>
    <span class="nb">attr_reader</span> <span class="ss">:name</span><span class="p">,</span> <span class="ss">:headers</span>

    <span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="nb">name</span><span class="p">)</span>
      <span class="vi">@name</span> <span class="o">=</span> <span class="nb">name</span>
      <span class="vi">@http_method</span> <span class="o">=</span> <span class="ss">:get</span>
      <span class="vi">@headers</span> <span class="o">=</span> <span class="p">{}</span>
    <span class="k">end</span>

    <span class="k">def</span> <span class="nf">path</span><span class="p">(</span><span class="n">value</span> <span class="o">=</span> <span class="kp">nil</span><span class="p">)</span>
      <span class="n">value</span> <span class="p">?</span> <span class="vi">@path</span> <span class="o">=</span> <span class="n">value</span> <span class="p">:</span> <span class="vi">@path</span>
    <span class="k">end</span>

    <span class="k">def</span> <span class="nf">http_method</span><span class="p">(</span><span class="n">value</span> <span class="o">=</span> <span class="kp">nil</span><span class="p">)</span>
      <span class="n">value</span> <span class="p">?</span> <span class="vi">@http_method</span> <span class="o">=</span> <span class="n">value</span> <span class="p">:</span> <span class="vi">@http_method</span>
    <span class="k">end</span>

    <span class="k">def</span> <span class="nf">header</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
      <span class="vi">@headers</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span>
    <span class="k">end</span>
  <span class="k">end</span>
<span class="k">end</span>

<span class="n">config</span> <span class="o">=</span> <span class="no">MyGem</span><span class="o">::</span><span class="no">Configuration</span><span class="p">.</span><span class="nf">new</span>

<span class="n">config</span><span class="p">.</span><span class="nf">endpoint</span> <span class="ss">:users</span> <span class="k">do</span>
  <span class="n">path</span> <span class="s2">"/api/v1/users"</span>
  <span class="n">http_method</span> <span class="ss">:get</span>
  <span class="n">header</span> <span class="s2">"Accept"</span><span class="p">,</span> <span class="s2">"application/json"</span>
<span class="k">end</span>

<span class="n">config</span><span class="p">.</span><span class="nf">endpoint</span> <span class="ss">:create_user</span> <span class="k">do</span>
  <span class="n">path</span> <span class="s2">"/api/v1/users"</span>
  <span class="n">http_method</span> <span class="ss">:post</span>
  <span class="n">header</span> <span class="s2">"Content-Type"</span><span class="p">,</span> <span class="s2">"application/json"</span>
<span class="k">end</span>

<span class="n">config</span><span class="p">.</span><span class="nf">endpoints</span><span class="p">.</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="nb">name</span><span class="p">,</span> <span class="n">ep</span><span class="o">|</span>
  <span class="nb">puts</span> <span class="s2">"</span><span class="si">#{</span><span class="nb">name</span><span class="si">}</span><span class="s2">: </span><span class="si">#{</span><span class="n">ep</span><span class="p">.</span><span class="nf">http_method</span><span class="p">.</span><span class="nf">upcase</span><span class="si">}</span><span class="s2"> </span><span class="si">#{</span><span class="n">ep</span><span class="p">.</span><span class="nf">path</span><span class="si">}</span><span class="s2">"</span>
  <span class="nb">puts</span> <span class="s2">"  headers: </span><span class="si">#{</span><span class="n">ep</span><span class="p">.</span><span class="nf">headers</span><span class="p">.</span><span class="nf">inspect</span><span class="si">}</span><span class="s2">"</span>
<span class="k">end</span></code></pre></div></div>
<div class="playground-controls">
<div class="playground-controls-buttons">
<button class="playground-run" disabled="">Run</button>
<button class="playground-reset" disabled="">Reset</button>
</div>
<span class="playground-controls-note">loads ~35MB Ruby environment</span>
</div>
<div class="playground-output"><pre><code>users: GET /api/v1/users
  headers: {"Accept" =&gt; "application/json"}
create_user: POST /api/v1/users
  headers: {"Content-Type" =&gt; "application/json"}</code></pre></div>
</div>

<p>DSLs feel elegant when they fit the domain. They also add cognitive load—users have to learn your DSL’s syntax and quirks, and you have to maintain the DSL code, which is never as simple as it looked when you started. Use this when the configuration genuinely benefits from the expressiveness; don’t use it just because you saw it in a conference talk.</p>

<h2 id="multi-source-configuration">Multi-source configuration</h2>

<p>Production applications often configure gems from multiple sources: environment variables, YAML files, and code. Handling this gracefully means establishing clear precedence rules.</p>

<pre><code class="language-ruby">module MyGem
  class Configuration
    attr_accessor :api_key, :timeout, :environment

    def initialize
      @timeout = 30
      @environment = :production
      load_from_env
    end

    def load_from_env
      @api_key = ENV["MY_GEM_API_KEY"] if ENV["MY_GEM_API_KEY"]
      @timeout = ENV["MY_GEM_TIMEOUT"].to_i if ENV["MY_GEM_TIMEOUT"]
      @environment = ENV["MY_GEM_ENV"]&amp;.to_sym if ENV["MY_GEM_ENV"]
    end

    def load_from_yaml(path)
      return unless File.exist?(path)

      yaml = YAML.load_file(path, permitted_classes: [Symbol])
      env_config = yaml[@environment.to_s] || yaml["default"] || {}

      @api_key ||= env_config["api_key"]
      @timeout = env_config["timeout"] if env_config["timeout"]
    end
  end

  class &lt;&lt; self
    def configure
      yield(configuration)
      configuration.load_from_yaml("config/my_gem.yml") if defined?(Rails)
    end
  end
end
</code></pre>

<p>The precedence here is: code overrides YAML overrides ENV overrides defaults. You could argue for different orderings—some gems prefer ENV to override everything for 12-factor compliance. Pick one and document it clearly. Users will still be confused, but at least you’ll have something to point them to.</p>

<p>A YAML file might look like this:</p>

<pre><code class="language-yaml"># config/my_gem.yml
default:
  timeout: 30

development:
  api_key: "sk-dev"
  timeout: 60

production:
  timeout: 15
</code></pre>

<p>Credentials in YAML files are a security concern. For anything sensitive, ENV variables or Rails credentials are safer choices. If you do support YAML credentials, at least check that the file permissions aren’t world-readable.</p>

<h2 id="choosing-a-pattern">Choosing a pattern</h2>

<p>Start with the simplest pattern that meets your needs:</p>

<ul>
  <li>Module accessors for gems with one or two settings;</li>
  <li>block-based configuration for most everything else;</li>
  <li>validation when misconfiguration causes confusing errors downstream;</li>
  <li>nested configuration when you have distinct subsystems that warrant their own namespace;</li>
  <li>per-instance configuration when users need multiple clients with different credentials;</li>
  <li>Rails integration when your gem is Rails-focused and you want to seem professional;</li>
  <li>DSLs when block-based genuinely isn’t expressive enough;</li>
  <li>multi-source when your users deploy to environments where ENV-based configuration is expected</li>
</ul>

<p>Most gems do fine with block-based configuration and validation. The fancier patterns exist for gems that genuinely need them—not as a badge of sophistication.</p>
]]></content>
    
    <summary>Configuration is often the first thing users interact with in your gem. It’s also the part most gem authors spend the least time thinking about—which explains a lot. This post covers the patterns you’ll encounter and implement, ordered from the simplest to the most involved.
</summary>
    
    
    <category term="ruby"/>
    
  </entry>
  
  <entry>
    <title>Creating a conventional, Stripe-like API with Grape and Ruby on Rails</title>
    <link href="/posts/conventional-api-with-grape-and-ruby-on-rails/" rel="alternate" type="text/html"/>
    <id>/posts/conventional-api-with-grape-and-ruby-on-rails/</id>
    <published>2023-03-24T00:00:00+00:00</published>
    <updated>2023-03-24T00:00:00+00:00</updated>
    <content type="html"><![CDATA[<p>Here’s the thing about API endpoints: you end up writing the same boilerplate over and over. Pagination metadata. Object type annotations. Wrapping collections in <code>data</code> arrays. Every endpoint looks identical except for the one line that matters—the actual query.</p>

<p>I got tired of it. So I built a convention system with Grape that handles all of it automatically. Now my endpoint code looks like this:</p>

<pre><code class="language-ruby"># thing.rb
get do
  pagy(Author.all)
end
</code></pre>

<p>And the response looks like this:</p>

<pre><code class="language-json">{
  "data": [
    {
      "id": 1,
      "name": "joshmn",
      "object": "author"
    }
  ],
  "has_more": true,
  "object": "list"
}
</code></pre>

<p>No manual JSON construction. No remembering to add <code>has_more</code>. No repetitive blueprint rendering calls. The framework handles it.</p>

<h2 id="why-grape-and-why-not-just-rails-controllers">Why Grape (and why not just Rails controllers)</h2>

<p>Before we get into the implementation—yes, you can do all of this with plain Rails controllers. You can also write your own HTTP server in C. The question is whether it’s worth your time.</p>

<p>Grape gives you things out of the box that Rails controllers don’t:</p>

<ul>
  <li>Typed parameters with automatic validation. Define <code>params { requires :id, type: Integer }</code> and Grape rejects bad input before your code runs. In Rails, you’re writing <code>params[:id].to_i</code> and hoping for the best;</li>
  <li>Built-in API versioning. Path-based, header-based, whatever you want. Rails has none of this;</li>
  <li>Documentation generation. Grape integrates with grape-swagger to generate OpenAPI specs from your parameter definitions. Your docs stay in sync with your code because they <em>are</em> your code;</li>
  <li>Custom formatters that let you intercept every response and transform it before it goes out the door</li>
</ul>

<p>Is Grape slower than raw Rails? Yes—maybe 10-20% on micro-benchmarks. Does that matter for your API that spends 95% of its time waiting on the database? Almost certainly not.</p>

<p>If you’re building something where that overhead matters, you probably shouldn’t be using Ruby at all.</p>

<h2 id="the-setup">The setup</h2>

<p>New app, basic dependencies:</p>

<pre><code class="language-shell">rails new stripeish
cd stripeish
bundle add blueprinter grape pagy
rails db:create
rails g model Author name
rails db:migrate
</code></pre>

<p>I use Blueprinter for serialization. You don’t have to—any serializer works—but I do. Pagy handles pagination.</p>

<h2 id="the-serialization-layer">The serialization layer</h2>

<p>First, an application-wide blueprint that adds the <code>object</code> field automatically:</p>

<pre><code class="language-ruby"># app/blueprints/application_blueprint.rb
class ApplicationBlueprint &lt; Blueprinter::Base
  def self.object_field
    field :object do |object|
      object.model_name.singular
    end
  end
end
</code></pre>

<p>Then a blueprint for each model:</p>

<pre><code class="language-ruby"># app/blueprints/author_blueprint.rb
class AuthorBlueprint &lt; ApplicationBlueprint
  object_field
  field :id
  field :name
end
</code></pre>

<p>Calling <code>object_field</code> in each blueprint is intentional. You’ll want some resources that don’t include the object type—nested associations, for example. Making it opt-in keeps you in control.</p>

<h2 id="the-api-structure">The API structure</h2>

<p>The main API class:</p>

<pre><code class="language-ruby"># app/api/api.rb
module API
  class API &lt; Grape::API
    version 'v1', using: :path
    prefix :api
    format :json

    mount ::V1::Authors
  end
end
</code></pre>

<p>Mount it in your routes:</p>

<pre><code class="language-ruby"># config/routes.rb
Rails.application.routes.draw do
  mount ::API::API =&gt; '/'
end
</code></pre>

<p>A basic resource:</p>

<pre><code class="language-ruby"># app/api/v1/authors.rb
module V1
  class Authors &lt; Grape::API
    namespace :authors do
      get do
        Author.all
      end
    end
  end
end
</code></pre>

<p>Hit <code>/api/v1/authors</code> and you’ll get… a raw array. Not what we want. Time for the interesting part.</p>

<h2 id="the-custom-formatter">The custom formatter</h2>

<p>Grape’s formatter is where the magic happens. It intercepts every response and transforms it before sending. We’ll use it to:</p>

<ol>
  <li>Automatically serialize ActiveRecord objects using their corresponding blueprints</li>
  <li>Inject pagination metadata when present</li>
  <li>Wrap collections in the Stripe-style <code>data</code> / <code>has_more</code> / <code>object</code> structure</li>
</ol>

<pre><code class="language-ruby"># app/api/custom_json_formatter.rb
class CustomJSONFormatter
  DATA_OBJECTS = [
    ActiveRecord::AssociationRelation,
    ActiveRecord::Relation,
    ActiveRecord::Base
  ].freeze

  EXCLUDED_OBJECTS = [String, Hash].freeze

  class &lt;&lt; self
    def call(resource, env)
      return resource.to_json if excluded?(resource)
      return resource.to_json unless serializable?(resource)

      blueprint = infer_blueprint(resource)
      options = extract_options(env)

      if collection?(resource)
        build_list_response(blueprint, resource, options, env)
      else
        blueprint.render(resource)
      end
    end

    private

    def excluded?(resource)
      EXCLUDED_OBJECTS.any? { |klass| resource.is_a?(klass) }
    end

    def serializable?(resource)
      DATA_OBJECTS.any? { |klass| resource.is_a?(klass) }
    end

    def collection?(resource)
      resource.is_a?(ActiveRecord::Relation) ||
        resource.is_a?(ActiveRecord::AssociationRelation)
    end

    def infer_blueprint(resource)
      model_class = collection?(resource) ? resource.klass : resource.class
      "#{model_class.name}Blueprint".constantize
    end

    def extract_options(env)
      options = {}
      if (pagy = env['api.pagy'])
        options[:has_more] = pagy.next.present?
      end
      options
    end

    def build_list_response(blueprint, resource, options, env)
      data = blueprint.render_as_hash(resource)

      response = {
        object: 'list',
        data: data
      }

      if (pagy = env['api.pagy'])
        response[:has_more] = pagy.next.present?
      end

      response.to_json
    end
  end
end
</code></pre>

<p>Wire it into your API:</p>

<pre><code class="language-ruby"># app/api/api.rb
module API
  class API &lt; Grape::API
    version 'v1', using: :path
    prefix :api
    format :json
    formatter :json, CustomJSONFormatter

    mount ::V1::Authors
  end
end
</code></pre>

<p>Now returning <code>Author.all</code> from an endpoint automatically serializes it using <code>AuthorBlueprint</code>, wraps it in a list structure, and includes the object type. No extra code required.</p>

<h2 id="pagination-that-doesnt-pollute-your-endpoints">Pagination that doesn’t pollute your endpoints</h2>

<p>Returning a tuple from every endpoint—<code>[pagy_object, records]</code>—is gross. The endpoint should return what it’s querying, not bookkeeping objects.</p>

<p>Instead, we’ll store pagination state in the Rack environment and let the formatter retrieve it:</p>

<pre><code class="language-ruby"># app/api/api.rb (inside the class)
helpers do
  include Pagy::Backend

  def pagy(collection)
    page = params[:page] || 1
    per_page = params[:per_page] || 30
    pagy_obj, items = super(collection, items: per_page, page: page)
    env['api.pagy'] = pagy_obj
    items
  end
end
</code></pre>

<p>The <code>env</code> hash is the Rack environment—available throughout the request lifecycle. We stash the Pagy object there, and the formatter picks it up later.</p>

<p>Now calling <code>pagy(Author.all)</code> paginates the collection <em>and</em> ensures the response includes <code>has_more</code>. The endpoint code doesn’t change.</p>

<h2 id="reusable-pagination-parameters">Reusable pagination parameters</h2>

<p>Define them once in a Grape DSL extension:</p>

<pre><code class="language-ruby"># config/initializers/grape_pagination.rb
module Grape
  module DSL
    module Parameters
      def pagination_params
        optional :page, type: Integer, desc: 'Page number (defaults to 1)'
        optional :per_page, type: Integer, desc: 'Results per page (defaults to 30, max 100)'
      end
    end
  end
end
</code></pre>

<p>Use them in any endpoint:</p>

<pre><code class="language-ruby">module V1
  class Authors &lt; Grape::API
    namespace :authors do
      params do
        pagination_params
      end
      get do
        pagy(Author.all)
      end
    end
  end
end
</code></pre>

<p>Grape generates parameter documentation automatically. Your API docs now include pagination params with descriptions.</p>

<h2 id="the-complete-picture">The complete picture</h2>

<p>Here’s everything together. The endpoint:</p>

<pre><code class="language-ruby">module V1
  class Authors &lt; Grape::API
    namespace :authors do
      params do
        pagination_params
      end
      get do
        pagy(Author.all)
      end

      route_param :id do
        get do
          Author.find(params[:id])
        end
      end
    end
  end
end
</code></pre>

<p><code>GET /api/v1/authors?per_page=2&amp;page=1</code> returns:</p>

<pre><code class="language-json">{
  "object": "list",
  "data": [
    { "id": 1, "name": "Alice", "object": "author" },
    { "id": 2, "name": "Bob", "object": "author" }
  ],
  "has_more": true
}
</code></pre>

<p><code>GET /api/v1/authors/1</code> returns:</p>

<pre><code class="language-json">{
  "id": 1,
  "name": "Alice",
  "object": "author"
}
</code></pre>

<p>No serialization calls. No response building. No pagination metadata juggling. The convention handles it.</p>

<h2 id="when-this-breaks-down">When this breaks down</h2>

<p>This approach has tradeoffs. Be aware of them:</p>

<p>Non-standard responses. Sometimes you need to return something that doesn’t fit the convention—aggregated data, custom structures, responses from external services. The formatter checks for <code>Hash</code> and <code>String</code> and passes them through unchanged, but you’ll need to build those responses manually.</p>

<p>Nested associations. If you want <code>Author</code> to include their <code>Books</code>, you need to handle that in the blueprint. The formatter won’t automatically serialize nested relations—you’ll define that relationship in <code>AuthorBlueprint</code> using Blueprinter’s association DSL.</p>

<p>Performance at scale. Inferring the blueprint class via <code>constantize</code> on every request adds overhead. It’s negligible for typical loads, but if you’re serving thousands of requests per second, you might want to cache the lookup or make the blueprint explicit.</p>

<p>Debugging opacity. When something goes wrong in the formatter, the stack trace isn’t always obvious. Add logging liberally while developing.</p>

<h2 id="what-you-get">What you get</h2>

<p>A system where adding a new resource means:</p>

<ol>
  <li>Create the model</li>
  <li>Create a blueprint with <code>object_field</code> and the fields you want</li>
  <li>Create a resource class with the query</li>
</ol>

<p>Everything else—serialization, pagination, response structure, documentation—is handled. Your endpoint code stays focused on the actual business logic.</p>

<p>That’s the whole point of conventions. You make decisions once, encode them in infrastructure, and stop thinking about them. The formatter is ~50 lines of code. It saves you from writing the same 10 lines in every endpoint, forever.</p>

<p>The full code is straightforward to extend. Add <code>total_count</code> to list responses. Add rate limiting headers. Add request timing. Whatever you need—you add it once, and every endpoint gets it.</p>

<p>That’s the leverage.</p>
]]></content>
    
    <summary>Here’s the thing about API endpoints: you end up writing the same boilerplate over and over. Pagination metadata. Object type annotations. Wrapping collections in data arrays. Every endpoint looks identical except for the one line that matters—the actual query.
</summary>
    
    
    <category term="ruby"/>
    
  </entry>
  
  <entry>
    <title>A plea for administration</title>
    <link href="/posts/a-plea-for-administration/" rel="alternate" type="text/html"/>
    <id>/posts/a-plea-for-administration/</id>
    <published>2022-12-27T00:00:00+00:00</published>
    <updated>2022-12-27T00:00:00+00:00</updated>
    <content type="html"><![CDATA[<p>Here’s the thing about admin panels: nobody wants to build them, nobody wants to maintain them, and the people who need them most are the ones with the least power to demand them.</p>

<p>This is a problem.</p>

<p>I ran a platform once. Not a big one, but big enough to have customer service people who weren’t me. From day one, I gave them tools—real tools. They could reset passwords, grant trials, see debugging info, approve orders. They could actually solve problems without filing a ticket and waiting for me to wake up.</p>

<p>Was this risky? Maybe. But I worked with these people. I trusted them. And more importantly, I trusted the audit trail. Every action was logged. If someone went rogue, I’d know. Nobody ever did.</p>

<p>The result: customers got help in minutes instead of hours. Support staff felt empowered instead of helpless. And I wasn’t the single point of failure for every minor issue.</p>

<p>I learned this philosophy in restaurants.</p>

<h2 id="the-restaurant-that-got-it-right">The restaurant that got it right</h2>

<p>For a period of time I worked in food service. 15 restaurants to be exact—not simultaneously, but still. Yes, someone with autism worked customer-facing in the service industry. Ask me how well that went. (Actually, don’t. I’m much happier now.)</p>

<p>One restaurant in particular taught me something I still carry. This place was massive—6 floors, 500 seats. Bad shit happens constantly in restaurants: drinks get screwed up, food takes too long, orders come out wrong. At most restaurants, front-line staff can’t do anything about it. Want to comp a meal? Find a manager. Want to offer a gift card? Find a manager. Want to actually fix the problem for the guest who’s sitting right in front of you, visibly annoyed? Find a manager.</p>

<p>The problem: at a 6-floor restaurant, the manager is never on your floor. They’re three floors away dealing with a different crisis.</p>

<p>This restaurant did something radical. They gave every server the ability to comp meals, issue discounts, and hand out gift cards. No manager approval required. The audit trail existed—it’s not like anyone was getting away with fraud—but the person responsible for the guest was actually empowered to be responsible for the guest.</p>

<p>It seems obvious when you say it out loud.</p>

<h2 id="the-pattern-i-keep-seeing">The pattern I keep seeing</h2>

<p>I don’t run my own thing anymore. I work for somebody. And I’ve noticed a pattern at every company since: there isn’t an investment in tools.</p>

<p>The workflow looks like this:</p>

<ol>
  <li>Customer reports a problem</li>
  <li>Support person logs into some bare-bones Grafana dashboard</li>
  <li>Support person can’t actually do anything useful</li>
  <li>Support person files a ticket for a developer</li>
  <li>Developer goes and looks at error logs</li>
  <li>Developer maybe opens a production REPL to inspect actual data</li>
  <li>Developer relays the diagnosis back to support</li>
  <li>Support finally tells the customer what’s going on</li>
</ol>

<p>That’s a lot of steps. That’s a lot of latency. That’s a lot of developer time spent on something a support person could have handled if they had the right interface.</p>

<h2 id="but-we-have-retool">“But we have Retool”</h2>

<p>Yes, there are tools trying to solve this. Retool, Appsmith, various low-code/no-code platforms. They’re fine. I’ve used them. They work for some workflows.</p>

<p>I still prefer <a href="https://opensourcerails.org/open-source-ruby-on-rails-apps-using-activeadmin-gem">ActiveAdmin</a> or <a href="https://opensourcerails.org/open-source-ruby-on-rails-apps-using-trestle-gem">Trestle</a> for Rails apps. They’re ugly. They’re opinionated. They’re also extremely fast to set up and they leverage your existing models and validations.</p>

<p>Here’s the uncomfortable part: most admin work is just CRUD. Create, read, update, delete. You don’t need a beautiful React app. You don’t need a design system. You need a form that edits a database row and doesn’t let you break constraints.</p>

<h2 id="what-admin-shouldnt-be">What admin shouldn’t be</h2>

<p>Admin should not be an extension of whatever stack you’re trying to push on customers.</p>

<p>I’ve seen companies with beautiful, polished customer-facing products and admin panels that literally don’t exist. Or admin that’s “log into the database and run SQL.” Or admin that requires a developer to make any change to any record.</p>

<p>That’s not acceptable. That’s a choice to prioritize engineering aesthetics over operational reality.</p>

<p>If your support team can’t reset a password without filing a Jira ticket, you don’t have a support team. You have a ticket-filing team.</p>

<h2 id="what-admin-should-be">What admin should be</h2>

<p>Quick. Dirty. Just enough.</p>

<p>Admin should be:</p>

<ul>
  <li>Fast to build. Days, not weeks. Use a framework. Generate scaffolds. Ugly is fine;</li>
  <li>Auditable. Every action logged. Who did what, when, to which record;</li>
  <li>Permission-aware. Not everyone needs access to everything. But the people who need access should actually have it;</li>
  <li>Operational, not aspirational</li>
</ul>

<p>The restaurant got this right. The people closest to the problem should be able to solve the problem. The audit trail is what protects you, not the bottleneck.</p>

<h2 id="the-real-cost">The real cost</h2>

<p>When you don’t invest in admin, the cost doesn’t show up on a balance sheet. It shows up in:</p>

<ul>
  <li>Slower response times for customers;</li>
  <li>Support staff who feel powerless and burn out;</li>
  <li>Developers pulled into operational work instead of building;</li>
  <li>Customers who leave because their problem took three days to resolve instead of three minutes</li>
</ul>

<p>You can’t measure the deals you lost because support couldn’t help fast enough. But they happened.</p>

<h2 id="where-to-start">Where to start</h2>

<p>If you’re sitting on a codebase with no admin tooling, here’s what I’d do:</p>

<ol>
  <li>
    <p><strong>Pick a framework and commit.</strong> For Rails: ActiveAdmin or Trestle. For Django: django-admin (it’s built in). For Node: AdminJS. Don’t build from scratch.</p>
  </li>
  <li>
    <p><strong>Start with read-only.</strong> Just let support see what’s in the database. Customer info, order history, subscription status. No editing yet. This alone solves half the ticket volume.</p>
  </li>
  <li>
    <p><strong>Add the high-frequency operations.</strong> What are the three things support asks developers to do most often? Password resets? Refunds? Status changes? Build those first.</p>
  </li>
  <li>
    <p><strong>Log everything.</strong> Every action, every user, every timestamp. This is how you earn trust. This is how you protect yourself when something goes wrong.</p>
  </li>
  <li>
    <p><strong>Expand based on real need.</strong> Don’t build speculatively. Wait until support says “I wish I could do X” and then build X.</p>
  </li>
</ol>

<p>The restaurant had a 500-person dining room and a handful of managers. They scaled by trusting their servers with real power and real accountability.</p>

<p>Your admin panel is the same problem. Trust your people. Give them tools. Keep the audit trail.</p>

<p>That’s how you manage at scale.</p>
]]></content>
    
    <summary>Here’s the thing about admin panels: nobody wants to build them, nobody wants to maintain them, and the people who need them most are the ones with the least power to demand them.
</summary>
    
    
    <category term="personal"/>
    
  </entry>
  
  <entry>
    <title>Database constraints first, validations second</title>
    <link href="/posts/database-constraints-first-validations-second/" rel="alternate" type="text/html"/>
    <id>/posts/database-constraints-first-validations-second/</id>
    <published>2022-11-24T00:00:00+00:00</published>
    <updated>2022-11-24T00:00:00+00:00</updated>
    <content type="html"><![CDATA[<p>A few years ago I inherited a codebase where the <code>users</code> table had no unique constraint on <code>email</code>. The model had <code>validates :email, uniqueness: true</code>, so everything seemed fine—until I found 847 duplicate email addresses in production.</p>

<p>Someone had written a rake task. It bypassed ActiveRecord. The model validation never ran, and the database didn’t care. Nearly a thousand users couldn’t log in because the system kept finding the wrong account.</p>

<p>This is the kind of bug that makes you question your career choices.</p>

<h2 id="the-conventional-approach">The conventional approach</h2>

<p>Rails tutorials teach you to think about validation first. You write <code>validates :email, presence: true, uniqueness: true</code> in your model, run your tests, and move on. Maybe you add database constraints later if you remember.</p>

<p>This creates a false sense of security. The model validation is a suggestion. It only runs when you use ActiveRecord the normal way. Console sessions, rake tasks, raw SQL, bulk imports, that legacy microservice someone built in 2019—none of these care about your model validations.</p>

<p>When the database is treated as dumb storage, your data integrity depends on every piece of code going through the front door. That’s not a realistic assumption.</p>

<h2 id="why-the-database-must-lead">Why the database must lead</h2>

<p>The database is the only layer that sees every write. It doesn’t matter if you’re using ActiveRecord, raw SQL, a different ORM, or some other app entirely. If you insert a row, the database constraints are checked.</p>

<p>This is why constraints like <code>NOT NULL</code>, unique indexes, and foreign keys exist. They’re not convenience features—they’re guarantees. A unique index on <code>email</code> means there will never be duplicate emails, period. No amount of buggy application code can create them.</p>

<p>When you put constraints in the database first, you’re making a statement about what your data must look like. Not what it should look like if everything goes right, but what it must look like regardless of how it got there.</p>

<p>Foreign keys are another obvious example. Without them, you can end up with <code>posts</code> pointing to <code>users</code> that don’t exist. With them, the database refuses to let that happen. You don’t have to remember to add <code>dependent: :destroy</code> or handle orphaned records—the problem can’t occur.</p>

<h2 id="the-cost-of-bad-data">The cost of bad data</h2>

<p>I want to be clear about what’s at stake here, because “data integrity” sounds abstract until you’ve lived with the alternative.</p>

<p>When your data can be in impossible states, every piece of code has to account for that. You write <code>user&amp;.email</code> instead of <code>user.email</code> because you’ve learned the hard way that sometimes <code>user_id</code> points to nothing. You add <code>rescue</code> blocks around code that shouldn’t be able to fail. You check for nil in places where nil shouldn’t exist. The codebase accumulates defensive scar tissue.</p>

<p>Queries get weird. You can’t just <code>SELECT * FROM orders WHERE user_id = ?</code> because some of those orders belong to deleted users and you’ll get crashes downstream. So you add <code>INNER JOIN users ON users.id = orders.user_id</code> everywhere, or you add <code>WHERE user_id IN (SELECT id FROM users)</code>, and now every query is slower and more complicated than it needs to be.</p>

<p>Debugging becomes archaeology. A bug report comes in: “this user can’t see their orders.” You check the orders table. The orders exist. You check the user. The user exists. You spend an hour before realizing the user_id on the orders is pointing to a different user who was deleted and whose ID got recycled. Or the email is duplicated across two accounts and they’re logged into the wrong one. Or the status field contains “shiped” instead of “shipped” because someone fat-fingered a console command three years ago.</p>

<p>Migrations become terrifying. You want to add <code>null: false</code> to a column that should never have been nullable. But it’s been nullable for two years, so now there’s garbage data in there. You have to write a one-off script to fix the existing rows before you can add the constraint—or worse, someone puts the data fix in the migration itself, and now your migrations are time bombs that behave differently depending on when they run. Sometimes you can’t even figure out what the correct value should be—the information is just gone.</p>

<p>New features get harder. You want to add a “subscription tier” to users. Simple enough, except you discover that 3% of your users have invalid subscription records, so now you have to handle that edge case in the new feature. And the next feature. And the one after that. The bad data is a tax on every piece of work you do.</p>

<p>The worst part is that none of this is visible until it’s too late. The app works fine when the data is clean. Tests pass because test data is clean. It’s only production—with years of accumulated writes from buggy code, rake tasks, console sessions, and that one time someone “fixed” something with raw SQL—where the impossible states live.</p>

<p>Every constraint you skip is a bet that nothing will ever write bad data through any path, forever. That’s a bad bet.</p>

<h2 id="why-you-still-need-model-validations">Why you still need model validations</h2>

<p>So why not just skip model validations entirely? Let the database constraints handle everything?</p>

<p>Because catching errors at the database level is miserable.</p>

<p>When a unique index violation happens, you get <code>ActiveRecord::RecordNotUnique</code>. When a NOT NULL constraint fails, you get <code>ActiveRecord::NotNullViolation</code>. These are exceptions, not validation errors. Your controller code has to rescue them. Your form doesn’t show a nice error message—the user sees a 500 page or a generic error.</p>

<p>Model validations exist for user experience. They let you catch problems before the database query happens, return friendly error messages, and populate <code>errors</code> so forms can highlight which fields have issues.</p>

<pre><code class="language-ruby">class User &lt; ApplicationRecord
  validates :email, presence: true, uniqueness: true
end

user = User.new(email: nil)
user.valid? # =&gt; false
user.errors[:email] # =&gt; ["can't be blank"]
</code></pre>

<p>This is much better than rescuing exceptions and trying to parse error messages to figure out what went wrong.</p>

<h2 id="the-rule-mirror-dont-diverge">The rule: mirror, don’t diverge</h2>

<p>Here’s the principle: put constraints in the database first, then add model validations that mirror those constraints.</p>

<p>If the database says NOT NULL, the model says <code>presence: true</code>. If there’s a unique index, the model says <code>uniqueness: true</code>. If there’s a foreign key, the model has the association. They should match.</p>

<pre><code class="language-ruby"># migration
class CreateUsers &lt; ActiveRecord::Migration[7.1]
  def change
    create_table :users do |t|
      t.string :email, null: false
      t.string :username, null: false
      t.timestamps
    end

    add_index :users, :email, unique: true
    add_index :users, :username, unique: true
  end
end

# model
class User &lt; ApplicationRecord
  validates :email, presence: true, uniqueness: true
  validates :username, presence: true, uniqueness: true
end
</code></pre>

<p>The model should never be more permissive than the database. If the model allows null emails but the database doesn’t, you’ll get exceptions in production. If the model allows duplicates but the database doesn’t, same thing.</p>

<p>You can make the model more restrictive—maybe you validate email format at the model level even though the database just has a NOT NULL. That’s fine. The database guarantees minimum integrity; the model can add business rules on top.</p>

<p>The problem is divergence. When the model says one thing and the database says another, you’re setting up future bugs. Someone will wonder why their save failed with a database exception when <code>valid?</code> returned true. Or worse, they’ll work around it in some creative way that makes the data model even more confusing.</p>

<h2 id="beyond-the-basics">Beyond the basics</h2>

<p>The simple cases—NOT NULL and unique indexes—are obvious. But databases can enforce more than that, and the same principle applies: constrain it at the database level, then mirror it in the model.</p>

<h3 id="check-constraints-for-allowed-values">Check constraints for allowed values</h3>

<p>Status columns are a classic example. You have an order that can be <code>pending</code>, <code>processing</code>, <code>shipped</code>, or <code>cancelled</code>. Without a constraint, someone will inevitably insert <code>shiped</code> or <code>Pending</code> or an empty string.</p>

<pre><code class="language-ruby"># migration
class CreateOrders &lt; ActiveRecord::Migration[7.1]
  def change
    create_table :orders do |t|
      t.references :user, null: false, foreign_key: true
      t.string :status, null: false, default: "pending"
      t.decimal :total, precision: 10, scale: 2, null: false
      t.timestamps
    end

    add_check_constraint :orders, "status IN ('pending', 'processing', 'shipped', 'cancelled')", name: "orders_status_check"
    add_check_constraint :orders, "total &gt;= 0", name: "orders_total_non_negative"
  end
end

# model
class Order &lt; ApplicationRecord
  STATUSES = %w[pending processing shipped cancelled].freeze

  belongs_to :user

  validates :status, presence: true, inclusion: { in: STATUSES }
  validates :total, presence: true, numericality: { greater_than_or_equal_to: 0 }
end
</code></pre>

<p>The check constraint guarantees the data. The model validation gives you <code>errors[:status]</code> with a message like “is not included in the list” instead of a database exception.</p>

<p>Note the <code>total &gt;= 0</code> constraint. You’d be surprised how often negative totals show up when there’s no constraint. Some bug in the discount calculation, a race condition in a refund flow, whatever. The database won’t let it happen.</p>

<h3 id="composite-unique-indexes">Composite unique indexes</h3>

<p>Sometimes uniqueness depends on scope. A user can have one subscription per plan, but could subscribe to multiple different plans. Or slugs need to be unique within an account, not globally.</p>

<pre><code class="language-ruby"># migration
class CreateSubscriptions &lt; ActiveRecord::Migration[7.1]
  def change
    create_table :subscriptions do |t|
      t.references :user, null: false, foreign_key: true
      t.references :plan, null: false, foreign_key: true
      t.datetime :expires_at
      t.timestamps
    end

    add_index :subscriptions, [:user_id, :plan_id], unique: true
  end
end

# model
class Subscription &lt; ApplicationRecord
  belongs_to :user
  belongs_to :plan

  validates :user_id, uniqueness: { scope: :plan_id, message: "already has this subscription" }
end
</code></pre>

<p>The composite index enforces that the combination is unique. The model validation mirrors it with <code>scope:</code> so you get a sensible error message.</p>

<h3 id="foreign-keys-with-cascading-behavior">Foreign keys with cascading behavior</h3>

<p>Foreign keys do more than prevent orphaned records. They can define what happens when the parent is deleted.</p>

<pre><code class="language-ruby"># migration
class CreateComments &lt; ActiveRecord::Migration[7.1]
  def change
    create_table :comments do |t|
      t.references :post, null: false, foreign_key: { on_delete: :cascade }
      t.references :user, null: false, foreign_key: { on_delete: :nullify }
      t.text :body, null: false
      t.timestamps
    end
  end
end

# model
class Comment &lt; ApplicationRecord
  belongs_to :post
  belongs_to :user, optional: true

  validates :body, presence: true
end
</code></pre>

<p>When a post is deleted, its comments are automatically deleted—no need for <code>dependent: :destroy</code> callbacks that might not run during bulk deletes. When a user is deleted, their comments stick around but <code>user_id</code> becomes null—preserving the content while removing the association.</p>

<p>The model reflects this: <code>belongs_to :user, optional: true</code> because the database allows null there after a cascade.</p>

<h3 id="partial-indexes-for-conditional-uniqueness">Partial indexes for conditional uniqueness</h3>

<p>Sometimes you only want uniqueness to apply in certain conditions. Active records should be unique, but you don’t care about archived ones.</p>

<pre><code class="language-ruby"># migration
class AddSlugToProjects &lt; ActiveRecord::Migration[7.1]
  def change
    add_column :projects, :slug, :string, null: false
    add_column :projects, :archived, :boolean, null: false, default: false

    add_index :projects, [:account_id, :slug], unique: true, where: "archived = false", name: "index_projects_unique_slug_when_active"
  end
end

# model
class Project &lt; ApplicationRecord
  belongs_to :account

  validates :slug, presence: true
  validates :slug, uniqueness: { scope: :account_id }, unless: :archived?
end
</code></pre>

<p>The partial index only enforces uniqueness for non-archived projects. The model validation uses <code>unless: :archived?</code> to match. Archived projects can have duplicate slugs—maybe you want to reuse a slug after archiving the old project.</p>

<h2 id="what-about-validations-the-database-cant-express">What about validations the database can’t express?</h2>

<p>Not everything fits in a constraint. Email format validation, conditional requirements, cross-model validations—these can’t be expressed in most databases.</p>

<p>These belong in the model, or in form objects and command objects if the logic is complex enough. The key is being clear about what each layer does:</p>

<p>Database constraints handle data integrity. The shape of valid data, regardless of how it’s written.</p>

<p>Model validations mirror those constraints for better error handling, plus add business logic that only applies to normal application flow.</p>

<p>Form and command objects handle context-specific validation. Maybe a user can be created without a phone number through one flow but not another. That’s not data integrity—that’s workflow logic.</p>

<p>Trying to cram everything into check constraints gets ugly fast. A check constraint for email format is possible in Postgres, but now you’re maintaining regex in SQL and hoping it matches what your application expects. Usually not worth it.</p>

<h2 id="closing">Closing</h2>

<p>The mental model is simple: database constraints are the contract, model validations are the friendly error messages.</p>

<p>Write the migration first. Add the NOT NULLs, the unique indexes, the foreign keys. Then write the model validations that mirror them. When in doubt, the database constraint wins—if you’re not sure whether something should be nullable, make it NOT NULL in the database and see what breaks.</p>

<p>Your data will thank you. Or at least, you won’t spend a weekend figuring out why there are 847 duplicate email addresses in production.</p>
]]></content>
    
    <summary>A few years ago I inherited a codebase where the users table had no unique constraint on email. The model had validates :email, uniqueness: true, so everything seemed fine—until I found 847 duplicate email addresses in production.
</summary>
    
    
    <category term="ruby"/>
    
  </entry>
  
  <entry>
    <title>REST-only controllers</title>
    <link href="/posts/rest-only-controllers/" rel="alternate" type="text/html"/>
    <id>/posts/rest-only-controllers/</id>
    <published>2022-06-11T00:00:00+00:00</published>
    <updated>2022-06-11T00:00:00+00:00</updated>
    <content type="html"><![CDATA[<p>A few months into a project, I watched a developer add a <code>mark_complete</code> action to a <code>TasksController</code> that already had <code>archive</code>, <code>unarchive</code>, <code>assign</code>, <code>unassign</code>, <code>prioritize</code>, <code>move_up</code>, <code>move_down</code>, and <code>duplicate</code>. The routes file had so many <code>member</code> blocks it looked like a DSL for avoiding REST.</p>

<p>Each custom action did one thing, took maybe five lines of code, and was perfectly reasonable on its own. But collectively they’d turned the controller into a junk drawer. The <code>before_action</code> filters had grown into a decision tree. The tests were full of <code>post :archive, params: { id: task.id }</code> mixed with <code>patch :update, params: { id: task.id, task: { ... } }</code>. Nobody could remember which actions were idempotent.</p>

<p>This is how most Rails controllers end up. You start with clean CRUD, then add “just one more action” until you have a god controller with fifteen public methods and a <code>case</code> statement in the <code>authorize</code> method.</p>

<p>I’ve seen worse. The codebase where someone decided custom actions were too messy, so they routed everything through <code>update</code> with a query param:</p>

<pre><code class="language-ruby">def update
  case params[:do]
  when "complete" then @task.complete!
  when "archive" then @task.archive!
  when "assign" then @task.assign_to!(params[:user_id])
  when "prioritize" then @task.prioritize!(params[:priority])
  else @task.update!(task_params)
  end
end
</code></pre>

<p>Now you’ve got one action doing five different things, no way to authorize them separately, and URLs like <code>/tasks/123?do=archive</code>. The routes file looks clean, but you’ve just moved the mess somewhere harder to find.</p>

<p>There’s a better way: stop adding custom actions. Turn those actions into resources.</p>

<h2 id="the-mental-shift">The mental shift</h2>

<p>Here’s a <code>TasksController</code> with custom actions:</p>

<pre><code class="language-ruby">class TasksController &lt; ApplicationController
  before_action :set_task, except: [:index, :new, :create]

  def index; end
  def show; end
  def new; end
  def create; end
  def edit; end
  def update; end
  def destroy; end

  def complete
    @task.complete!
    redirect_to @task
  end

  def reopen
    @task.reopen!
    redirect_to @task
  end

  def archive
    @task.archive!
    redirect_to tasks_path
  end

  def unarchive
    @task.unarchive!
    redirect_to @task
  end

  # ... and on it goes
end
</code></pre>

<p>Routes:</p>

<pre><code class="language-ruby">resources :tasks do
  member do
    post :complete
    post :reopen
    post :archive
    post :unarchive
  end
end
</code></pre>

<p>Now here’s the same behavior with REST-only controllers:</p>

<pre><code class="language-ruby">class TasksController &lt; ApplicationController
  # Just the standard seven actions
end

class Tasks::CompletionsController &lt; ApplicationController
  def create
    @task = Current.user.tasks.find(params[:task_id])
    @task.complete!
  end

  def destroy
    @task = Current.user.tasks.find(params[:task_id])
    @task.reopen!
  end
end

class Tasks::ArchivalsController &lt; ApplicationController
  def create
    @task = Current.user.tasks.find(params[:task_id])
    @task.archive!
  end

  def destroy
    @task = Current.user.tasks.find(params[:task_id])
    @task.unarchive!
  end
end
</code></pre>

<p>Routes:</p>

<pre><code class="language-ruby">resources :tasks do
  scope module: :tasks do
    resource :completion
    resource :archival
  end
end
</code></pre>

<p>The insight is that “complete” isn’t an action you do to a task—it’s a resource you create. A completion. When you complete a task, you’re creating its completion. When you reopen it, you’re destroying that completion.</p>

<p>Same with archival, publication, subscription, assignment, and every other “action” you’ve been adding to controllers. They’re all resources.</p>

<h2 id="why-this-matters">Why this matters</h2>

<p>It’s not just aesthetics. REST-only controllers have practical benefits.</p>

<p>When I join a project, the first thing I open is <code>config/routes.rb</code>. It’s diagnostic. A routes file full of <code>member do</code> blocks and custom actions tells me exactly what’s waiting in the controllers: conditionals, god objects, authorization spaghetti. A routes file that’s just nested resources tells me the controllers are probably clean too. The routes file is a leading indicator for the whole codebase.</p>

<p>The controllers stay small. The <code>Tasks::CompletionsController</code> above is eight lines. It does one thing: manage the completion state of a task. There’s no room for it to grow into a mess because there’s nothing else it could do. You can’t add a <code>prioritize</code> action to it—that would be absurd. The constraint forces you to create a new controller for the next concept.</p>

<p>Authorization becomes obvious. When every controller handles one resource, you can authorize at the controller level without complex conditionals.</p>

<pre><code class="language-ruby">class Tasks::CompletionsController &lt; ApplicationController
  before_action :ensure_can_complete_task

  def create
    @task.complete!
  end

  def destroy
    @task.reopen!
  end

  private

  def ensure_can_complete_task
    head :forbidden unless Current.user.can_complete?(@task)
  end
end
</code></pre>

<p>No <code>case</code> statement in your <code>authorize</code> method. No checking which action is being called. The controller handles completions, so you check completion permission. Done.</p>

<p>Routes become predictable. <code>POST /tasks/123/completion</code> creates a completion. <code>DELETE /tasks/123/completion</code> removes it. Anyone who knows REST can guess your URLs. You don’t need to document that “completing a task is a POST to <code>/tasks/:id/complete</code>” because that’s a custom action—you use the standard verbs on a standard resource.</p>

<p>Tests get consistent too. Every controller test uses the same HTTP verbs for the same purposes. <code>post</code> creates something. <code>delete</code> removes something. <code>patch</code> updates something. You’re not mixing <code>post :archive</code> with <code>patch :update</code> in the same test file.</p>

<h2 id="patterns">Patterns</h2>

<p>After doing this for a while, you start to see the same patterns everywhere.</p>

<h3 id="boolean-states-become-singular-resources">Boolean states become singular resources</h3>

<p>If something can be turned on or off, it’s a singular resource. Creating it turns it on; destroying it turns it off.</p>

<pre><code class="language-ruby">resource :completion     # complete/reopen
resource :archival       # archive/unarchive
resource :publication    # publish/unpublish
resource :subscription   # subscribe/unsubscribe
resource :pin            # pin/unpin
resource :lock           # lock/unlock
</code></pre>

<p>The controller is always the same shape:</p>

<pre><code class="language-ruby">class Posts::PublicationsController &lt; ApplicationController
  def create
    @post.publish!
  end

  def destroy
    @post.unpublish!
  end
end
</code></pre>

<p>Use <code>resource</code> (singular) because there’s only one publication per post. The routes are <code>/posts/:post_id/publication</code>, not <code>/posts/:post_id/publications</code>.</p>

<h3 id="many-to-many-relationships-become-plural-resources">Many-to-many relationships become plural resources</h3>

<p>If a user can watch multiple posts, and a post can be watched by multiple users, that’s a plural resource.</p>

<pre><code class="language-ruby">resources :posts do
  scope module: :posts do
    resources :watches, only: [:create, :destroy]
  end
end
</code></pre>

<pre><code class="language-ruby">class Posts::WatchesController &lt; ApplicationController
  def create
    @post.watches.create!(user: Current.user)
  end

  def destroy
    @post.watches.find_by!(user: Current.user).destroy!
  end
end
</code></pre>

<p>Or if you don’t want the intermediate model:</p>

<pre><code class="language-ruby">class Posts::WatchesController &lt; ApplicationController
  def create
    @post.watch_by(Current.user)
  end

  def destroy
    @post.unwatch_by(Current.user)
  end
end
</code></pre>

<p>Either way, the controller is just managing the watch relationship.</p>

<h3 id="attribute-updates-become-singular-resources">Attribute updates become singular resources</h3>

<p>Sometimes you want to update a single attribute with its own UI and permissions. Don’t route it through the main controller’s <code>update</code>—make it a resource.</p>

<pre><code class="language-ruby">resource :role, only: [:edit, :update]
</code></pre>

<pre><code class="language-ruby">class Users::RolesController &lt; ApplicationController
  before_action :ensure_admin

  def edit
  end

  def update
    @user.update!(role: params[:role])
  end
end
</code></pre>

<p>This is cleaner than checking <code>if params[:role].present?</code> in <code>UsersController#update</code> and having different authorization rules for role changes versus profile changes.</p>

<h3 id="position-changes-become-create-only-resources">Position changes become create-only resources</h3>

<p>Moving something up or down in a list? That’s creating a new position.</p>

<pre><code class="language-ruby">resource :up_position, only: :create
resource :down_position, only: :create
</code></pre>

<pre><code class="language-ruby">class Items::UpPositionsController &lt; ApplicationController
  def create
    @item.move_up!
  end
end

class Items::DownPositionsController &lt; ApplicationController
  def create
    @item.move_down!
  end
end
</code></pre>

<p>You’re not updating the item—you’re creating an up-position or down-position for it. The naming might feel weird at first, but it’s consistent with REST. You’re always creating or destroying something.</p>

<h3 id="context-specific-actions-become-namespaced-controllers">Context-specific actions become namespaced controllers</h3>

<p>Sometimes the same action has different behavior in different contexts. Archiving a message from the inbox versus archiving it from a thread view might have different side effects—one might mark the whole thread as read, the other might not.</p>

<pre><code class="language-ruby">namespace :inbox do
  resources :messages do
    scope module: :messages do
      resource :archival
    end
  end
end

namespace :threads do
  resources :messages do
    scope module: :messages do
      resource :archival
    end
  end
end
</code></pre>

<pre><code class="language-ruby">class Inbox::Messages::ArchivalsController &lt; ApplicationController
  def create
    @message.archive!
    @message.thread.mark_as_read!
  end
end

class Threads::Messages::ArchivalsController &lt; ApplicationController
  def create
    @message.archive!
  end
end
</code></pre>

<p>The namespace communicates intent. This isn’t just <code>Messages::ArchivalsController</code>—the context is right there in the class name. You can have different authorization, different side effects, different everything.</p>

<h2 id="common-objections">Common objections</h2>

<h3 id="but-now-i-have-fifty-controllers">“But now I have fifty controllers.”</h3>

<p>Yes. And? Each one is ten lines long, does exactly one thing, and is trivial to understand. Would you rather have five controllers with 200 lines each?</p>

<p>The filesystem is free. Your ability to hold complexity in your head is not.</p>

<h3 id="the-urls-look-weird">“The URLs look weird.”</h3>

<p>Do they? <code>POST /tasks/123/completion</code> seems pretty clear. You’re creating a completion for task 123. Compare to <code>POST /tasks/123/complete</code>—why is that verb better than a noun?</p>

<p>If anything, the resource-based URLs are more RESTful. REST is about resources and representations, not about mapping domain verbs onto HTTP verbs.</p>

<h3 id="its-more-files-to-navigate">“It’s more files to navigate.”</h3>

<p>Your editor has fuzzy search. Cmd+P “completions controller” and you’re there. Meanwhile, finding the <code>complete</code> action in a 300-line <code>TasksController</code> requires scrolling or searching.</p>

<h3 id="some-actions-dont-fit-the-createdestroy-pattern">“Some actions don’t fit the create/destroy pattern.”</h3>

<p>Usually they do if you think about it differently.</p>

<ul>
  <li>“Bump priority” becomes creating a <code>priority_bump</code></li>
  <li>“Send reminder” becomes creating a <code>reminder</code></li>
  <li>“Recalculate totals” becomes creating a <code>recalculation</code></li>
  <li>“Sync with external service” becomes creating a <code>synchronization</code></li>
</ul>

<p>If it really doesn’t fit—maybe it’s a pure query with no side effects—consider whether it belongs in a controller at all. Maybe it’s a view concern, or a separate endpoint that returns JSON for your frontend to handle.</p>

<h3 id="what-about-wizards-or-multi-step-forms">“What about wizards or multi-step forms?”</h3>

<p>Each step is a resource. Step one creates the draft. Step two updates it with more data (or creates a “step two completion”). The final step creates the actual record from the draft. Or use a form object—but that’s orthogonal to controller design.</p>

<h2 id="the-constraint-that-liberates">The constraint that liberates</h2>

<p>This approach feels restrictive at first. You want to add a quick action and instead you have to create a new controller, a new route, maybe a new view directory. It seems like ceremony.</p>

<p>But the ceremony is the point. The friction of creating a new controller makes you think about what you’re actually modeling. Is this really a new concept, or am I just being lazy? Often it is a new concept that deserves its own home.</p>

<p>The constraint also prevents the gradual accumulation of mess. You can’t add “just one more action” because there’s no place to put it. You have to create something new, and creating something new feels like a bigger decision than appending to something existing. That’s good. It should feel like a decision.</p>

<p>After a while, you stop thinking about it. Actions are resources. Every controller handles exactly one resource with exactly the standard seven actions (or fewer). The routes file is a clean tree of nested resources. Each controller is small enough to read in one screenful.</p>

<p>And you never have to look at a fifteen-action controller with a <code>case</code> statement in the authorization method again.</p>
]]></content>
    
    <summary>A few months into a project, I watched a developer add a mark_complete action to a TasksController that already had archive, unarchive, assign, unassign, prioritize, move_up, move_down, and duplicate. The routes file had so many member blocks it looked like a DSL for avoiding REST.
</summary>
    
    
    <category term="ruby"/>
    
  </entry>
  
</feed>
