Untrod

LLM-Powered Django Admin Fields

2024-04-14T00:00:00-07:00

Note: All relevant code can be found in the corresponding Github repo.

I work at Grove Collaborative -- an ecommerce company that sells thousands of different products. We use Django and the Django admin as our product information system to manage, among other things, all of the copy on our Product Detail Pages (PDPs). In order to more efficiently and consistently write copy for different PDP sections, we integrated ChatGPT directly into the Django Admin.

While not quite encapsulated enough to make it a standalone Django plugin, the code is generic enough to quickly integrate ChatGPT with any Django text field, and provides a number of affordances to Django admin users to modify and test the underlying prompts.

What we did

In short, we now have little buttons next to Django Admin text fields that will generate copy according to a specific prompt and data payload. For example, we have a section on our PDPs titled "Why we believe in this product", backed by an attribute on the Product model called why_we_believe. In the Django Admin, when viewing a Product model, there is a button that our merchandising team can use to populate the field. The text is editable and reviewable; ChatGPT provides initial copy, but humans are still very much in the loop.

Each "LLM-enabled" text field is mapped to a specific prompt and bespoke data payload that provides the model-specific context to ChatGPT.

We also have prompts and buttons for fields like "review summary" (summarizes the pros and cons of a product based on customer reviews), "hero ingredients", and more. The underlying prompts are editable in the admin so our merchants and tweak the output without engineering involvement:

Finally, a playground area allows admin users to test the prompts out without leaving Django:

We can quickly add GPT-powered copywriting to any text field, on any model, in the Django Admin. The relevant code is available in a Github repo, with some instructions on how to get started. The balance of this post simply explicates the code and patterns.

How it works

Note that some of the code below is slightly simplified from what's in the repo in order to aid comprehension.

First up, a simple model to store and manage the prompts that will power each text field:

class GPTPrompt(CoreModel):
    system_message_template = models.TextField(blank=False)
    human_message_template = models.TextField(blank=False)
    key = models.CharField(blank=False, unique=True, max_length=100)
    model = models.IntegerField(choices=GPT_MODELS, default=GPT3)

    def prompt(self):
        return ChatPromptTemplate.from_messages(
            [
                SystemMessagePromptTemplate.from_template(self.system_message_template),
                HumanMessagePromptTemplate.from_template(self.human_message_template),
            ]
        )

    @property
    def model_name(self):
        return GPT_MODELS[self.model][1]

    def __str__(self):
        return self.key

After registering the model in the Django admin, the prompts are now editable.

There is a bit of magic to the "key" value, which maps to a hard-coded value on a class that knows how to inject the proper contextual data for a given prompt. For example, the prompt in the screenshot above needs to know how to get product_json. So we write a little class that can provide that data (one class for each prompt):

class BelievePromptData(object):
    PROMPT_KEY = "WHY_WE_BELIEVE_PROMPT"

    def __init__(self, model_id):
        self.product = Product.objects.get(pk=model_id)

    def __call__(self):
        keys = ["name", "selling_points", "manufacturer_description", "ingredients"]
        return {"product_json": json.dumps({k: str(getattr(self.product, k)) for k in keys})}

A single Django view will handle all of the client-side text-generation requests coming from the admin buttons:

def generic_llm_prompt_view(request, promptDataCls=None):
    prompt = GPTPrompt.objects.get(key=promptDataCls.PROMPT_KEY)
    prompt.prompt_data = promptDataCls(request.GET.get("model_id"))
    return HttpResponse(json.dumps({"text": prompt.get_text()}), content_type="application/json")

The correct PromptData class is injected when registering the URLs:

from functools import partial

urlpatterns += [
    re_path(r"^generate-believe-text/", partial(generic_llm_prompt_view, promptDataCls=BelievePromptData), name="generate-believe-text"),
]

Note in the repo, there are also URLs registered for the playground.

To recap:

A request comes in to e.g. generate-believe-text/?model_id=123
The generic llm_prompt_view passes the model_id into the PromptData object specified in the URL endpoint (in this case BelievePromptData)
BelievePromptData gets the relevant model and turns it into the JSON blob that we want to inject into our prompt. The key of the PromptData class maps to the correct GPTPrompt object.
The GPTPrompt gets loaded, merges in the JSON blob, and calls out to ChatGPT to generate the text.

This pattern makes it really easy to add more prompts; an admin user can create a new prompt (and test it out in the playground), and an engineer adds a corresponding PromptData class and new URL.

Hooking it up to the Django Admin UI

Lastly, we have to add the actual button next to the relevant field. Torturing the Django admin is never particularly fun, so we created a widget you can stick next to arbitrary model text fields, and map to the different URLs, corresponding to the various PromptData classes. The Widget is simple:

class LLMTextAreaWidget(Textarea):
    template_name = "widgets/llm_text_area_widget.html"

    def __init__(self, req_url, *args, **kwargs):
        self.llm_req_url = req_url
        super().__init__(*args, **kwargs)

    def get_context(self, name, value, attrs):
        context = super().get_context(name, value, attrs)
        context["widget"]["llm_req_url"] = self.llm_req_url
        context["widget"]["btn_class"] = f"{name}-generate-btn"
        return context

The corresponding JavaScript is kind of a mess -- but functional. The HTML and JS for the widget can be found in the complete repo.

Then simply add the widget to the admin form. E.g.

class ProductForm(forms.ModelForm):

    class Meta:
        model = Product
        fields = "__all__"
        widgets = {
            "why_we_believe": LLMTextAreaWidget(req_url="/llm/generate-believe-text/"),
        }

And voila! Our merchandisers can click the button to generate text and pre-populate the field. This fields remain a 'normal' text field; it's still overridable by the admin user. Merchants often use the buttons to get started, and then tweak the copy from there.

Robot Dad

2023-11-26T00:00:00-08:00

See discussion of this post on Hacker News

Tired of Alexa's feeble "here's what I found on wikianswerspam.com" responses to my eight-year-old son's science questions, I whipped up Robot Dad during my Thanksgiving break. He now runs in the background of our family computer.

Robot Dad sounds like real dad, thanks to voice cloning from Eleven Labs (very easy; I rambled about Formula 1 into a new MacOS Voice Memo for about sixty seconds, uploaded it, and voila), and answers appropriately for an eight-year-old, while deflecting prank requests -- though I suspect prompt injection will soon be second nature to this generation.

Here's my son, Dash, interacting with Robot Dad.

The delays are real, and the speech-to-text is only so-so, but it manages to be just good enough to clear the "provides value" bar. Robot Dad will also inject context from the previous question into the prompt. Dash could follow up with "Robot Dad, tell me more about it" and ChatGPT would know what to do.

A few dozen lines of code glues together different AI services, for a remarkable result. The wakeword and speech-to-text happen locally, while the AI response (ChatGPT) and text-to-speech are via HTTP. It's trivial to move the LLM bit to a local Llama2 instance, but I haven't found a satisfactory text-to-speech model that can do voice cloning locally.

I also made a quick speech visualization (turns out kids are not very engaged reading console log messages) that of course ended up providing more entertainment value than the ENTIRE MODERN MIRACLE OF ARTIFICIAL INTELLIGENCE. Code for the visualization is in this gist.

Code for Robot Dad is below. You will need API keys for Picovoice (and a wakeword), Eleven Labs, and OpenAI. You can pick a pre-existing Eleven Labs voice or clone your own.

import os, json, threading, time
import pvporcupine, pvcheetah
from pvrecorder import PvRecorder
from elevenlabs import voices, generate, play, stream
import openai

ENDPOINT_DURATION_SECONDS = 2 # 'Quiet' seconds indicating the end of audio capture
VOICE = 'Dad' # Via Eleven Labs
AUDIO_DEVICE_NAME = 'MacBook Pro Microphone'
AUDIO_DEVICE = PvRecorder.get_available_devices().index(AUDIO_DEVICE_NAME)
OPENAI_MODEL = 'gpt-3.5-turbo-1106'

BASE_PROMPT = """You are Robot Dad, and will be speaking with one of my children,
trying to be a helpful parent. You explain things at a level appropriate for
an eight-year-old.

You are encouraging and helpful, but won't tolerate any inappropriate requests
or attempts at pranks or jokes. If you are asked or told anything
inappropriate, you gently say "nice try - but Robot Dad isn't falling for that!"

If you don't know how to reply, simply say "I'm just Robot Dad, not real dad -
so I'm afraid I can't help you with that".

You usually answer in no more than 4 sentences - kids do not have long attention
spans - but you can provide longer answers if it's clearly needed.
"""

PREV_CTX_PROMPT = """

The last request and response you received is below. The next request may or may
not be a continuation of this conversation.

Previous request:
%s

Previous response:
%s
"""

PREV_CTX_TIMEOUT = 60 # seconds

keyword_paths=['%s/wakewords/Robot-Dad.ppn' % ROOT]

porcupine_key = os.environ.get("PORCUPINE_API_KEY")
openai.api_key = os.environ.get("OPENAI_API_KEY")

porcupine = pvporcupine.create(
    access_key=porcupine_key,
    keyword_paths=keyword_paths)

cheetah = pvcheetah.create(
    access_key=porcupine_key,
    endpoint_duration_sec=ENDPOINT_DURATION_SECONDS,
    enable_automatic_punctuation=True)

recorder = PvRecorder(
    frame_length=porcupine.frame_length,
    device_index=AUDIO_DEVICE)

break_audio = generate(text="Got it! Robot Dad is thinking...", voice=VOICE)
alert_audio = generate(text="What's up kiddo?", voice=VOICE)

def llm_req(prompt, txt):
    messages= [
        {"role": "system", "content": prompt},
        {"role": "user", "content": f'Here is what the child has said: {txt}'}
    ]

    resp = openai.ChatCompletion.create(
      model=OPENAI_MODEL,
      messages=messages
    )
    return resp['choices'][0]['message']['content']


# Speech-to-text using Picovoice's Cheetah
def capture_input():
    transcript = ''
    while True:
        partial_transcript, is_endpoint = cheetah.process(recorder.read())
        transcript += partial_transcript
        if is_endpoint:
            transcript += cheetah.flush()
            break
    return transcript


def play_async(audio):
    audio_thread = threading.Thread(target=play, args=(audio,))
    audio_thread.start()


def main():
    print('Listening...')

    recorder.start()

    prev_request = ''
    prev_response = ''
    last_wake_time = None

    try:
        while True:
            pcm = recorder.read()
            result = porcupine.process(pcm)

            if result >= 0:
                print('Detected Robot Dad')
                play_async(alert_audio)

                prompt = BASE_PROMPT
                current_time = time.time()
                if last_wake_time and current_time - last_wake_time < PREV_CTX_TIMEOUT:
                    prompt += PREV_CTX_PROMPT % (prev_request, prev_response)
                last_wake_time = current_time

                transcript = capture_input()
                print('Heard request: %s' % transcript)
                prev_request = transcript

                play_async(break_audio)

                resp = llm_req(prompt, transcript)
                print('Answering: %s' % resp)
                prev_response = resp

                resp_audio = generate(text=resp, voice=VOICE, stream=True)
                stream(resp_audio)
    except KeyboardInterrupt:
        pass

    recorder.stop()
    print('Stopped.')
main()

Copy Editing a Novel with ChatGPT

2023-06-28T00:00:00-07:00

My wife, a writer, recently and unexpectedly needed a final round of copy-editing on a manuscript she had written some time ago. There wasn't time to hire a professional, and even splitting it up, we couldn't get it done between the two of us fast enough.

Luckily, I know Python and have an OpenAI API key! Thus, I asked ChatGPT to copy-edit the book. I believed this was in ChatGPT's wheelhouse for a few reasons:

Copy-editing is about true errors; spelling, grammar, etc. So "chunks" of text can be copy-edited separately. The entire novel doesn't need to be in the model's context window. Searching for plot holes or timeline inconsistencies in a novel-length text would be another matter entirely.
ChatGPT has impeccable spelling and grammar. Therefore I assumed it would be good at catching errors in text.
It's easy to provide good examples. "Few shot" prompting tends to improve performance.

Below we'll look at the approach I took, and the results. I've included illustrative, but incomplete, code. Full code is available here. I used GPT4 as a coding assistant for much of this. And boy it's great. It's so nice to get working directory-traversal code (for instance) that does just what I need, without spending five minutes Googling and reading docs. GPT4-assisted Python scripting makes exercises like this much more enjoyable.

To preview the conclusion: ChatGPT performed much less well than I thought, but still provided a usable copy editing pass. It is significantly better than nothing, but surprisingly noisy and erratic.

Chunking up the text

Using some handy ChatGPT-provided code, I chunked the novel into separate text files of (in this case) at most 1000 words. This is well under the maximum GPT4 context size of 8000 tokens (1000 words is perhaps 1250 tokens), but I found that smaller chunks lead to more consistency in finding true errors.

Separate files made downstream experimentation easier (I could delete all but a few chunks; I could write resume code when the openai API failed, etc). The chunking is deterministic and fast.

Copy Editing with ChatGPT

Now we send the chunks to Openai and ChatGPT. The prompt below works...OK. I should have perhaps provided more rigorous few-shot examples. Nonetheless, I experimentally determined that the examples below did lend considerable consistency to the output. Without it, ChatGPT will come up with different ways of providing the feedback for each chunk, which makes the results hard for a human to process, and a machine to parse (more on that later). If I did this again, I'd spend another iteration or two on the prompt. I do believe there is quality juice to squeeze there.

You are a copy editor looking for issues in a novel before it is submitted to publishers. You are looking for obvious grammar and spelling issues and any information that is obviously incorrect. Do not make suggestions related to style, or edits for clarity. Just focus on copy errors.

Please format your responses as a series of bullet points. Start with a quote of a few words from the novel that you are copy-editing (so it's easy to find in the novel), then follow with your comments/corrections.

Here are some examples of good copy edits:

"an two hundred year" -> "a two hundred year"

"on to the veranda" -> "onto the veranda"

"full memory of the night" -> "full memories of the night"

"Her and Luis's dog" -> "Her and Luis' dog"

"Felicia stiffened almost indecipherably." -> "Felicia stiffened almost imperceptibly."

"The is the family kitchen." -> "This is the family kitchen."

Do not suggest substitutions of one type of punctuation mark for another. For example, do not suggest replacing ` with ', or “ with ".

Looping through each of the 104 chunks with GPT4 took about 35 minutes and cost just over $8 in API fees. In retrospect, for this task, I suspect GPT3.5 could give equivalent results more quickly, at less cost.

Initial Result

So...does it work? Not as well as I'd hoped. Perhaps 1/3 of the suggestions are legitimate suggestions/corrections. The others fall into one of three categories:

Non-corrections. The original text and the new text are identical, or simply changed a quote or apostrophe type.
- "I didn’t tell the police." -> "I didn't tell the police."
Oddly specific stylistic suggestions (in spite of the prompt asking to avoid these):
- "She yelped. Savvy tried to apologize" -> "She yelped, and Savvy tried to apologize"
Total hallucinations. This text, for instance, simply doesn't appear in the corresponding chunk:
- "highway ears as they passed" -> "highway ears as it passed"

Trying to read through an output riddled with false-positives is mentally exhausting. Which of these are legitimate suggestions and which contain no signal?

"bulletproof glass partition. “You" -> "bulletproof glass partition, "You"
"Detective Moorehead, please,” Savvy" -> "Detective Moorehead, please," Savvy"
"loudly banging each letter" -> "loudly, banging each letter"
"Finished. Straightened a stack" -> "Finished, straightened a stack"
"What is this in" -> "What is this in,"
"with Detective Moorehead,” she" -> "with Detective Moorehead," she"

It's like a vision test. Yikes.

After carefully reading chatGPT's output, I wrote a function to detect hallucinations (19 of them for those keeping score at home), non-corrections, and persnickety suggestions that simply involved swapping a backtick for an apostrophe (e.g. encoding nuances).

def is_real_correction(input_str):
    left_side = input_str.split('->')[0].strip(' -"').replace('’', "'").replace('“', '"').replace('”', '"')
    right_side = input_str.split('->')[1].strip(' -"').replace('’', "'").replace('“', '"').replace('”', '"')
    if left_side.endswith('"') and not right_side.endswith('"'):
        left_side = left_side[:-1]
    # full_novel_text check is for hallucinations
    if left_side != right_side and 'remove extra space' and left_side in full_novel_text:
        return f'- {left_side} -> {right_side}'

Final Result

For a novel of about 90,000 words, GPT4 identified 1120 "issues", which the script above reduced to 811. I suspect feeding the corrections back into ChatGPT with a prompt to identify only real corrections could have worked -- but good old-fashioned string manipulation with Python is also plenty good. I'm not yet reaching for ChatGPT to solve all of my problems, but perhaps I am a luddite.

Of the remaining corrections, about 25% are truly legitimate corrections, 50% are somewhat random wording and stylistic changes, and the remaining 25% are legitimate grammar issues, but are contained within spoken dialogue or are otherwise used purposefully. If this was a copy-edit for a magazine article, this last 25% would be useful, as the author would want the writing to conform strictly to a standard (e.g. Chicago Manual of Style) -- but is of limited utility in the context of a novel.

Here are some representative corrections.

Legitimate Copy Errors

Felicia stiffened almost indecipherably. -> Felicia stiffened almost imperceptibly.
The is the family kitchen. -> This is the family kitchen.
"No, doesn’t mean that" -> "No, it doesn't mean that"

Random Rephrasings

"He stood on the opposite side" -> "He stood on the other side"
"He put his glass down" -> "He set his glass down"
"the overhead lights flashed" -> "the overhead lights flickered"
"The corners of his mouth" -> "The corners of his lips"

Debatable Copy Corrections

"His laugh was contagious and his spirit big, I'm told.” -> "His laugh was contagious, and his spirit was big, I'm told.”
"Not exactly the apology Savvy was expecting" -> "Not exactly the apology Savvy expected"

Conclusion

Yeah - I mean - not bad? Kind of OK for $8? (Ignoring the ~2 hours of time it took me to wrangle the code)

I can't help but feel a little disappointed, as this felt like a slam-dunk for ChatGPT. As the old saying goes, AI is great at solving 80% of any problem.

With that said, I do think ChatGPT (or even a much simpler model) could be turned into an excellent copywriter through fine tuning. The tech is surely capable of doing much better. Given some more time, I would have loved to try that approach (especially because training data is easy to generate for this use case). Perhaps in a future blog post...

Short Stories for Engineers

2022-06-09T00:00:00-07:00

I've recently been reminded of a few favorite short stories, and discovered some new ones, that swirl around themes of existence, engineering. order, systems, computing, and information.

None of these are deep cuts; they are well known among sci-fi readers and short story lovers. But I thought I'd compile the ones I love nonetheless.

Isaac Asimov (1956) - The Last Question (and Answer)

The Last Question is Asmiov's best story, and everyone knows it (Asimov included). A later story, The Last Answer, is also excellent but related only indirectly, through an existential theme. It's worth noting that Answer by Fredric Brown is likely an (uncited) inspiration (it was written a few years prior). Lastly, a Douglas Adams fan will surely recognize the famous Last Question parody from Hitchiker's Guide.

Harlan Ellison (1965) - "Repent, Harlequin!", Said the Ticktockman

Nearly a period piece; bizarre and subverts expectations in terms of writing and style, while having a plot that appears cliche, except that the cliche originated with this story. In other words, purely original, but the ideas have entered collective consciousness such that they seems well worn.

"Repent, Harlequin!", Said the Ticktockman

Ted Chiang (2014) - Exhalation

Boy oh boy Ted Chiang is exciting. Story of Your Life is another short of his that provided the plot of the excellent movie Arrival. Apart from that, this is likely his best-known story, and my favorite.

Exhalation

Roald Dahl (1954) - The Great Automatic Grammatizator

Particularly interesting in the context of GPT-3 et al.

The Great Automatic Grammatizator

Andy Weir (2009) - The Egg

Confession: This is not actually one of my favorites. It's nominally as thought-provoking as any other story here, but it just doesn't stick in my mind the same way as some others. Nonethless, it's enjoyable and so thematically close to some of the other stories that I had to include it in this list.

The Egg

Kurt Vonnegut (1961) - Harrison Bergeron

Perhaps you've heard of the Handicapper General? It's hard to believe this was written in 1961. In fact, it's a good reminder that we are prisoners of our own experience and societal issues that may seem novel are not new at all.

Harrison Bergeron

Guidepost - Business Value

2022-04-09T00:00:00-07:00

We write code to deliver business value. We should therefore select solutions where most of the investment is going into the differentiated bits that drive business value.

This means we want to write more code that is only relevant to Grove (because it uniquely solves our business problems) and less code that is related to general scaling or infrastructure challenges. If we can buy solutions to the other problems (or just side-step them), let’s do that, and take advantage of other customers of that 3rd party solution subsidizing the cost for us.

Guidepost - Getting to Production

2022-04-09T00:00:00-07:00

We’ve always prioritized getting to production frequently. Lots of goodness flows from shipping early and often. The problems associated with shipping too frequently are so much easier to fix than the problems of shipping too infrequently.

When releases are infrequent, there is a natural tendency to “shrink to fit”; to cut a corner or create pressure to hit a release date, lest your code has to wait until the next release. Even a release calendar like e.g. “Release once per day, except Friday” is one bad Thursday release away from 5 days between releases — enough to create artificial pressure that works directly against all efforts to ship quality software.

Getting to production frequently means that releases are no big deal, and reverts are no big deal.

That’s not to say “more releases are always better” — we are already at a point where it’s impractical to manage when code is being released, and it’s challenging to determine the root cause of new exceptions because releases are so frequent. But the appropriate rate of releases is likely higher than intuition would suggest.

Guidepost - How Long Should Software Last?

2022-04-09T00:00:00-07:00

At each stage of the business, there is a correspondingly appropriate length of time you should plan for your code to last - the horizon of the software’s utility. In the very early stages of a start-up, code may only need to last a few days. “We’ll try it, and if it works, we can rewrite it next week!”.

And for a series B company, it doesn’t make sense to write software that will take the company public, but you also can’t have code breaking at 2x the scale and rewriting every few months.

In a late stage company (where Grove is today), your code should last for about 3 years. If it has to be rewritten within 12 months, then it is not robust enough. If it breaks in 2-3 years due to growth and/or changes in the business, then so be it. You did your job. But of course, this is a floor on utility – not an ask for planned obsolescence. Obviously code that costs the same to create, and lasts longer, is always preferable.

You don’t need to solve for scale problems that aren’t within the current horizon.

Note that experiments are entirely different. We may write a feature experimentally, which, if it works, will need to be re-written to meet our production/lifetime horizon criteria. Earlier in the company’s history we (like many) had a poor track record of actually re-writing these things and would let hacky experiment code live for a long time. We don’t do that anymore. It’s frustrating for the business, to see the “B” variation of a test do really well, and then have to wait multiple sprints to see it scaled out to 100% of customers -- but that’s the price of operational code that has to last for multiple years.

Guidepost - Scaling Principles

2022-04-09T00:00:00-07:00

Distributed systems are hard. We should exhaust many options before the solution is “do something distributed”. This is also why we are fine creating services, but are not advocates of “micro services”.

Similarly, in most cases we should exhaust vertical scaling before moving to horizontal. It’s almost always less expensive to buy a larger box than it is to staff a complex distributed systems project. In a business like Grove’s, we should make prudent financial decisions with our tech choices, but ultimately the economic drivers of the business are elsewhere — not in technology OpEx. Note that other companies, mostly SaaS companies that are, on some level, reselling infrastructure; Mixpanel, Segment, etc, do have tech OpEx as a key part of cost structure. It’s just not the case here.

We believe that better code can address a lot of scaling challenges; it’s just a website selling soap, after all. While we are “at scale” from a revenue standpoint, we are relatively low-volume in the scheme of web applications, simply because of the size and access patterns of our customers. This is not to trivialize the complexity of scaling, but we believe that, at this stage, the answer is more likely to be “create a better data model with smarter caching” rather than “shard the database”.

Guidepost - Services & Monoliths

2022-04-09T00:00:00-07:00

Monoliths are not inherently problematic. Perhaps they are even good! Close coupling is the evil to avoid. With good design, monoliths can avoid tight coupling. Services can also be a valuable tool for achieving this. But the purpose of decreasing coupling is to decrease overall complexity. Therefore a service that decreases coupling, but increases overall complexity (e.g. introduces fiendish distributed systems problems), is to be avoided.

When we say “service” we really mean “a piece of software with a well-defined boundary”, which doesn’t necessarily mean “web service that ships and operates independently of the monolith”.

A more formal definition might be “functionality that is accessed using a prescribed interface and conforms to constraints and policies specified by a service description”. Note this definition does not say “and it needs a separate CI/CD pipeline”.

Services are also a management tool.

Conway’s Law is, broadly, that you ship your org chart; if you are building a compiler, and you have three teams working on it, you’ll get a three-pass compiler. Services are useful through this lens. They create boundaries between pieces of technology so that small teams can operate independently and have ownership of their results.

Create service boundaries when it’s useful to create ownership over a problem domain and when the service decreases net complexity.

Guidepost - Technology & Language Selection

2022-04-09T00:00:00-07:00

There is value in consistency, and fungibility (e.g. the extent to which a developer can work on multiple projects). There’s value in better bus factors.

Therefore, there is a high bar for selecting a new technology that the team doesn’t know.

At some level, programming languages are just syntax and don’t matter that much -- but the lift in a new language/stack’s ecosystem can be considerable, and beginning with something new makes beginners’ mistakes more likely.

We default to the things we already use, unless there is a strong reason that it won’t work.

Note this is different from iterative improvement; introducing a new library or a new pattern or a new practice is different than introducing an entirely new technology or language. We should absolutely be pushing to modernize our Python, JavaScript, SQL, etc. development practices; this is not a case for stagnation, but a case for uniformity-by-default.

Technologies don’t get selected simply because they are interesting.

Guidepost - Testing

2022-04-09T00:00:00-07:00

The primary value of great tests is that it allows developers (especially those who didn’t write the original code!) to refactor. With tests at your back, you can refactor with confidence. Without the ability to refactor, our velocity will slowly decrease to zero.

We love automated testing. If you have ever found yourself deploying code to production based solely on your automated tests, and realizing afterwards - “Oh sh**! I never actually tested this manually!”, then you are doing it right. It is impossible for developers to have confidence their code works without tests.

If you think of tests as “homework”, or something to be added at the end, once it’s working - keep working at automated testing! Tests are your buddy along the development journey, ensuring at each step along the way that your software works the way you expect. If that feeling is not deep in your bones, keep working at it! Train yourself to find the joy and confidence in writing tests as you go.

It’s a very normal state of affairs that your change will have more test code than application code.

As you move from unit tests to integration tests, you trade off diagnostic utility for coverage; unit tests cover little, but when they fail, it’s immediately clear what went wrong. Integration tests can cover a huge variety of regressions, but when they fail it’s harder to understand why. Use both.

The broadest tests of all are browser/device automation tests. They are there to ensure that nothing terrible is about to happen in production -- not to ensure correctness of your code.

We aren’t believers in test-driven-development; it’s a great way to never get anything done.

Adventures in Candy Land

2021-07-23T00:00:00-07:00

I play Candy Land occasionally with my two boys (the six year old gets it much more than the three year old). It has the strange property of involving both zero skill and zero agency. There are absolutely no decisions for the player to make at any point; the outcome of the game is determined as soon as the cards are shuffled.

This makes it reduce to a game that is about as fun as flipping a coin, but significantly more fun to hack together in Python and explore!

We'll do a rules refresher in a sec, but first, some simple classes representing the board, a player, and a "move" object, which doesn't affect gameplay, but will keep track of each turn so we can analyze games later.

class Board(object):

    def __init__(self, shortcuts, colors, specials, licorice, length):
        self.shortcuts = shortcuts
        self.colors = colors
        self.specials = list(specials.keys())
        self.licorice = licorice
        self.spaces = (self.colors * ((length//len(colors))+1))
        for k, v in specials.items():
            self.spaces.insert(v, k)
        self.spaces = self.spaces[:length]


class Player(object):

    def __init__(self, name):
        self.name = name
        self.position = -1


class Move(object):

    def __init__(self, player, card, new_position):
        self.player = player.name
        self.card = card
        self.new_position = new_position

Now we can create a variety of different Candy Land boards. Here's a board with full fidelity to the original game (which you can see here). For the morbidly curious, I got the right indices for the licorice, specials, etc by laying out the board in a spreadsheet.

LENGTH = 134
COLORS = list('RPYBOG')
SPECIALS = {
    'Plumpy': 8,
    'Mr. Mint': 17,
    'Jolly': 42,
    'Gramma Nut': 74,
    'Princess Lolly': 94,
    'Queen Frostine': 103
}
LICORICE = [47,85,120]
SHORTCUTS = {4:58, 33:46}

board = Board(SHORTCUTS, COLORS, SPECIALS, LICORICE, LENGTH)

def spc_fmt(i, s):
    if i in board.licorice:
        return 'Licorice: {}'.format(s)
    if i in board.shortcuts.keys():
        return 'Shortcut to {0}: {1}'.format(board.shortcuts[i], s)
    return s

print([spc_fmt(i, s) for i, s in enumerate(board.spaces)])

Here's our board with some formatting to signify licorice and shortcuts:

['R', 'P', 'Y', 'B', 'Shortcut to 58: O', 'G', 'R', 'P', 'Plumpy', 'Y', 'B', 'O', 'G', 'R', 'P', 'Y', 'B', 'Mr. Mint', 'O', 'G', 'R', 'P', 'Y', 'B', 'O', 'G', 'R', 'P', 'Y', 'B', 'O', 'G', 'R', 'Shortcut to 46: P', 'Y', 'B', 'O', 'G', 'R', 'P', 'Y', 'B', 'Jolly', 'O', 'G', 'R', 'P', 'Licorice: Y', 'B', 'O', 'G', 'R', 'P', 'Y', 'B', 'O', 'G', 'R', 'P', 'Y', 'B', 'O', 'G', 'R', 'P', 'Y', 'B', 'O', 'G', 'R', 'P', 'Y', 'B', 'O', 'Gramma Nut', 'G', 'R', 'P', 'Y', 'B', 'O', 'G', 'R', 'P', 'Y', 'Licorice: B', 'O', 'G', 'R', 'P', 'Y', 'B', 'O', 'G', 'Princess Lolly', 'R', 'P', 'Y', 'B', 'O', 'G', 'R', 'P', 'Queen Frostine', 'Y', 'B', 'O', 'G', 'R', 'P', 'Y', 'B', 'O', 'G', 'R', 'P', 'Y', 'B', 'O', 'G', 'Licorice: R', 'P', 'Y', 'B', 'O', 'G', 'R', 'P', 'Y', 'B', 'O', 'G', 'R', 'P']

Good stuff.

Ok, onto the game! For those of you who have not played Candy Land recently, you can read the official rules here, but the most salient bits are:

Players take turns drawing cards and then...do what the card says. Again...there is zero decision making on the part of the player.
In the original game (which is mirrored in the settings above), there are 66 cards; 6 "special character" cards, and 60 color cards. There are six colors, with eight "single" and two "double" cards in each color. The game code (which we'll look at next) generates the cards.
When a player draws a color card, move forward to the next color of that space. For double color cards, move to the next-next color of that space.
If the card is a "special character" card, go directly to that space (even if it is behind you).
There are three "licorice" spaces. If you land on one of these, you are stuck until you draw that color again. Note that in more recent versions the player simply loses a turn on these spaces. We are keeping with the old-school "stuck" rules.
There are two "shortcuts". If you land exactly on one of the shortcut spaces, immediately follow the shortcut.
You win by getting to the "end" of the board (e.g. drawing a color that does not appear in front of you).

With that in mind, here is the game code:

class Game(object):

    def __init__(self, players, board):
        self.cards = []
        self.board = board
        self.players = [Player(p) for p in players]

    def create_cards(self):
        return self.board.specials + self.board.colors*8\
               + [c+c for c in self.board.colors]*2

    def draw(self):
        import random
        if not self.cards:
            self.cards = self.create_cards()
            random.shuffle(self.cards)
        return self.cards.pop()

    def _gets_unstuck(self, card, position):
        return card not in self.board.specials and\
               card[:1] == self.board.spaces[position]

    def generate_move(self, player, card):
        if player.position in self.board.licorice:
            if not self._gets_unstuck(card, player.position):
                return Move(player, card, player.position)

        if card in self.board.specials:
            return Move(player, card, self.board.spaces.index(card))

        pos = player.position

        # Loop to handle 'double' color cards
        for x in card:
            try:
                pos = self.board.spaces.index(x, pos+1)
                if pos in self.board.licorice:
                    break
                pos = self.board.shortcuts.get(pos, pos)
            except ValueError as e:
                pos = None
                break

        return Move(player, card, pos)

    def _take_turn(self):
        # arguably "too clever by half" way of determining turns
        player = self.players[len(self.cards) % len(self.players)]
        m = self.generate_move(player, self.draw())
        player.position = m.new_position
        return m

    def play(self):
        moves = []
        while m := self._take_turn():
            moves.append(m)
            if m.new_position is None: break
        return moves

With default settings, the 66 cards of the deck are created in the create_cards method.

The core game logic is in generate_move: given a card and a player, we calculate the new position of the player. This is somewhat interesting and it's the algorithm that must be running inside my six-year-old's head, and the algorithm that my three-year-old can't quite get a handle on.

The play game loop simply takes turns until the win condition is met (which happens when the drawn card's position doesn't is not present in the remainder of the board - see line 40).

Now that we have our game objects, we can play (using the board we created previously):

players = ['Chris', 'Maggie']
g = Game(players, board)
moves = g.play()

We can print out the moves and see what happened. Weirdly, formating the move well was perhaps the hardest part of this entire exercise.

def fmt_move(i, m, b):
    if m.new_position is None:
        out = "On turn {0}, {1} draws {2} and WINS!"
    elif m.new_position in b.licorice:
        out = "On turn {0}, {1} draws {2} and is stuck on {3}."
    else:
        out = "On turn {0}, {1} draws {2} and moves to {3}."
    return out.format(
        i+1,
        m.player,
        m.card,
        m.new_position if m.new_position is None else m.new_position+1)

for i, m in enumerate(moves):
    print(fmt_move(i, m, g.board))

Here's the output, eliding over a number of moves for brevity:

On turn 1, Chris draws Plumpy and moves to 9.
On turn 2, Maggie draws O and moves to 59.
...
On turn 27, Chris draws P and is stuck on 48.
On turn 28, Maggie draws GG and is stuck on 121.
On turn 29, Chris draws RR and is stuck on 48.
On turn 30, Maggie draws R and moves to 127.
On turn 31, Chris draws R and is stuck on 48.
On turn 32, Maggie draws Princess Lolly and moves to 95.
...
On turn 49, Chris draws RR and moves to 89.
On turn 50, Maggie draws R and moves to 127.
On turn 51, Chris draws Y and moves to 91.
On turn 52, Maggie draws GG and WINS!

Grr, I lost. Oh well - I'm sure there will be more games...in fact...we can build a harness to play many games, and see the effect of the various board components:

def play_games(config, num):
    return [Game(**config).play() for _ in range(num)]

def analyze(name, results):
    from statistics import mean, median
    lengths = list(map(len, results))
    print("Turn stats for {0} '{1}' games:".format(len(lengths), name))
    print("Mean:    {}".format(mean(lengths)))
    print("Median:  {}".format(median(lengths)))
    print("Min/max: {0}/{1}".format(min(lengths), max(lengths)))
    print("")

PLAYERS = ['Chris']
NUM_GAMES = 10000
LENGTH = 134

STANDARD = {
    'players': PLAYERS,
    'board': Board(SHORTCUTS, COLORS, SPECIALS, LICORICE, LENGTH)}

NO_LICORICE = {
    'players': PLAYERS,
    'board': Board(SHORTCUTS, COLORS, SPECIALS, [], LENGTH)}

NO_SPECIALS = {
    'players': PLAYERS,
    'board': Board(SHORTCUTS, COLORS, {}, LICORICE, LENGTH)}

NO_SHORTCUTS = {
    'players': PLAYERS,
    'board': Board({}, COLORS, SPECIALS, LICORICE, LENGTH)}

analyze('Standard', play_games(STANDARD, NUM_GAMES))
analyze('No Licorice', play_games(NO_LICORICE, NUM_GAMES))
analyze('No Specials', play_games(NO_SPECIALS, NUM_GAMES))
analyze('No Shortcuts', play_games(NO_SHORTCUTS, NUM_GAMES))

Note that these games have only one player (me!) since they are for analytical purposes (vs. HARD-CORE COMPETITIVE purposes). The results:

Turn stats for 10000 'Standard' games:
Mean:    38.6897
Median:  33.0
Min/max: 4/242

Turn stats for 10000 'No Licorice' games:
Mean:    34.5284
Median:  30.0
Min/max: 4/178

Turn stats for 10000 'No Specials' games:
Mean:    27.6359
Median:  28.0
Min/max: 10/74

Turn stats for 10000 'No Shortcuts' games:
Mean:    40.6276
Median:  35.0
Min/max: 5/210

You now know a deep secret of the universe: that the 'special' cards in Candy Land on average send players backwards more often than forwards. My GOD! I always thought they were there to help! Doing some simple math, we determine:

Impact of licorice: 4.16 turns
Impact of specials: 11.05 turns
Impact of shortcuts: -1.94 turns

Shortcuts reduce games by an average of 1.9 turns, while licorice and specials add to the average lengths of games. Cool!

I, for one, thoroughly enjoyed acquiring this utterly worthless knowledge. There was never a chance of discovering a "better" way to play Candy Land as the player has zero agency. But I suppose that's the joy of Candy Land for small children; a perfectly even chance of winning against your far more sophisticated, Python-programming parents.

Explorations in The Card Game SET®

2021-06-28T00:00:00-07:00

SET is a classic pattern-matching card game that's been around since the 1970s, but really took off in the 90s. It hits all the right marks for a great game casual game; easy to learn, works for any number of players, you can play for 5 minutes or for hours, little equipment required, and it's always competitive.

Here's how it works: using a special set of 81 cards, deal 12 at random, face up. Each card is has a unique combination of 4 attributes: Color, shape, fill, and number, each of which has three different possible values. So after dealing you get something like this:

Players shout "set!" when they identify a "valid set". A valid set consists of exactly three cards, for which each of the four attributes are either all the same or all different. So here's a valid set from the deal above:

You can see that, for each of the four attributes, all the cards either completely match or are completely different.

Attribute	Card 1	Card 2	Card 3	Set
Number	3	2	1	Different
Fill	Lined	Open	Solid	Different
Color	Green	Red	Purple	Different
Shape	Squiggle	Squiggle	Squiggle	Same

If you were to change any attribute on any of the cards in the set, you'd have some attribute for which 2 cards matched, and the third different, thus making it an invalid set. When someone finds a set, you remove it, and deal three new cards in their place. You go through the entire deck until there are <=12 cards left, with no more sets. Whoever claimed the most sets wins.

The rules are very simple, but the fun comes from staring at the cards as they are dealt, frantically trying to find sets before you opponent. If my brain had a cooling fan, it would immediately turn on as my cognitive load instantly maxes out.

To understand a little better what I was looking for (and to fiddle around with Python, which is always fun), I sought to determine how many sets are likely to be in the first 12 dealt cards. Here's the code to generate a deck of SET cards, and detect whether a set (or multiple sets) are present. There are a few "unnecessary" methods in the below because (spoilers!) we are going to simulate a lot of Set games later and this turned out to make quite a difference.

> Run the code

You can run and edit all of the code in this blog post right from your browser.

from itertools import combinations
import random

attributes = ['number', 'fill', 'color', 'shape']

def make_cards():
    # 1, 2, 3, open, shaded, filled, purple, green, red, diamond, squiggle, oval
    return ['{}{}{}{}'.format(n, f, c, s)
            for n in '123' for f in 'osf' for c in 'pgr' for s in 'dso']

def _valid(elems):
    # Members of elems are all the same, or all different
    return len(set(elems)) != 2

def _is_set(a,b,c):
    return all([
        _valid([a[x], b[x], c[x]])
        for x in range(4)]) # 4 is the number of distinct attributes

def find_sets(cards):
    return [c for c in combinations(cards, 3) if _is_set(*c)]

def find_set(cards):
    # The same as above, but bails as soon as it finds a set. Again, performance thing.
    for c in combinations(cards, 3):
        if _is_set(*c):
            return c

def sets_exist(cards):
    # Perf optimization: 21+ cards guarantee a set
    # via https://mathscinet.ams.org/mathscinet-getitem?mr=2031694
    return len(cards) > 20 or bool(find_set(cards))

def test():
    assert len(make_cards()) == 81
    assert len(find_sets(make_cards())) == 1080

    cards = '2ors,2srd,3opd,3fpd,3fgs,2frs,2fpo,2sps,1opo,1sps,2fro,1fgd'.split(',')
    assert sets_exist(cards)
    assert not sets_exist(cards[:8])
    assert len(find_set(cards)) == 3
    assert len(find_sets(cards)) == 6

test()

We can now empirically determine some facts about the game. I simulated one million 12-card deals (from a fresh deck each time) to see how many sets appear in the initial draw of a game.

sets = [len(find_sets(random.sample(make_cards(), 12)))
        for _ in range(1000000)]

print("Avg sets:", sum(sets) / len(sets))
print("Min sets:", min(sets))
print("Max sets:", max(sets))

for x in range(min(sets), max(sets)+1):
    print("{} sets {:.4%} of the time ({} of {})".format(x, sets.count(x) / len(sets), sets.count(x), len(sets)))

Results:

Avg sets: 2.78542
Min sets: 0
Max sets: 12
0 sets 3.2228% of the time (32228 of 1000000)
1 sets 14.5164% of the time (145164 of 1000000)
2 sets 26.1262% of the time (261262 of 1000000)
3 sets 27.2253% of the time (272253 of 1000000)
4 sets 17.9905% of the time (179905 of 1000000)
5 sets 8.0349% of the time (80349 of 1000000)
6 sets 2.3360% of the time (23360 of 1000000)
7 sets 0.4585% of the time (4585 of 1000000)
8 sets 0.0731% of the time (731 of 1000000)
9 sets 0.0129% of the time (129 of 1000000)
10 sets 0.0031% of the time (31 of 1000000)
11 sets 0.0002% of the time (2 of 1000000)
12 sets 0.0001% of the time (1 of 1000000)

To build intuition, we can chart it:

%matplotlib notebook
import matplotlib.pyplot as plt
plt.plot([x for x in range(min(sets), max(sets)+1)],
         [sets.count(x) for x in range(min(sets), max(sets)+1)])
plt.show()

Neat! The > 10 set case turns out to be quite rare, even though in theory (see page 18) 14 sets can exist in a 12-card deal.

Next up, let's play a full game. The code below accomplishes that.

def run_game():

    deck = make_cards()
    random.shuffle(deck)
    table = deck[:12]
    deck = deck[12:]

    state_log = []

    while sets_exist(table + deck):
        s = find_set(table) or []

        state_log.append({'table': len(table), 'deck': len(deck), 'set': bool(len(s))})

        table = [c for c in table if c not in s]

        # Don't put more cards down if a set was drawn from a 15+ card table
        if not(s and len(table) >= 12):
            table += deck[:3]
            deck = deck[3:]

    return state_log

result = run_game()
for step in result:
    print("{table} cards on the table. {deck} cards in the deck. Set: {set}".format(**step))

Game output:

12 cards on the table. 69 cards in the deck. Set: True
12 cards on the table. 66 cards in the deck. Set: True
12 cards on the table. 63 cards in the deck. Set: True
12 cards on the table. 60 cards in the deck. Set: False
15 cards on the table. 57 cards in the deck. Set: True
12 cards on the table. 57 cards in the deck. Set: True
12 cards on the table. 54 cards in the deck. Set: True
12 cards on the table. 51 cards in the deck. Set: True
12 cards on the table. 48 cards in the deck. Set: True
12 cards on the table. 45 cards in the deck. Set: True
12 cards on the table. 42 cards in the deck. Set: True
12 cards on the table. 39 cards in the deck. Set: True
12 cards on the table. 36 cards in the deck. Set: True
12 cards on the table. 33 cards in the deck. Set: True
12 cards on the table. 30 cards in the deck. Set: True
12 cards on the table. 27 cards in the deck. Set: False
15 cards on the table. 24 cards in the deck. Set: True
12 cards on the table. 24 cards in the deck. Set: True
12 cards on the table. 21 cards in the deck. Set: True
12 cards on the table. 18 cards in the deck. Set: True
12 cards on the table. 15 cards in the deck. Set: True
12 cards on the table. 12 cards in the deck. Set: True
12 cards on the table. 9 cards in the deck. Set: True
12 cards on the table. 6 cards in the deck. Set: True
12 cards on the table. 3 cards in the deck. Set: True
12 cards on the table. 0 cards in the deck. Set: True
9 cards on the table. 0 cards in the deck. Set: True

Up next, let's see how many times in an average game there are no sets in 12 cards, and 15 cards are required (Or 18! Or, in theory at least, 21!). We simulate 10,000 games and keep track of how many times there are different cards on the table (as well as how many cards are on the table at the end game):

from collections import Counter

fifteens, eighteens, twentyones, final_cards_on_table = [], [], [], []

for _ in range (10000):
    result = run_game()
    table = [r['table'] for r in result]

    # Remove a final 3 cards if a final set has been found
    final = table[-1]-3 if result[-1]['set'] else table[-1]
    final_cards_on_table.append(final)

    fifteens.append(table.count(15))
    eighteens.append(table.count(18))
    twentyones.append(table.count(21))

print("Final cards on table:\n{}".format(Counter(final_cards_on_table)))
print("Fifteens: {} times in {} games.".format(sum(fifteens), sum([x>0 for x in fifteens])))
print("Eighteens: {} times in {} games.".format(sum(eighteens), sum([x>0 for x in eighteens])))
print("Twenty Ones: {} times in {} games.".format(sum(twentyones), sum([x>0 for x in twentyones])))

Let's see:

Final cards on table:
Counter({6: 4853, 9: 4838, 12: 172, 0: 136, 15: 1})
Fifteens: 14327 times in 6696 games.
Eighteens: 144 times in 139 games.
Twenty Ones: 0 times in 0 games.

It appears we'll get a "15" on the table at some point about 2/3rds of the time, and 18 cards only 1.4% of the time. It's also interesting that the vast majority (97%!) of games end with either 6 or 9 cards left on the table, and it's about evenly split between those two outcomes. This surprised me; it's not intuitive to me that the final 12 cards would have about the same odds of containing a set as a final 9 cards. Said differently, given a final 12 cards, it's even odds that it contains exactly 1 set or exactly 2 sets (non-intersecting). You'll also notice that "3" does not appear as endgame scenario. This makes sense - if 78 of 81 cards in the deck have been made into a set, the last three cards must also be a set.

Lastly, I wondered how useful it would be to continue looking for a set after your opponent calls "Set", but before they have collected the cards from the 12 on the table. We generate 100,000 first draws, find all sets, pick one at random, remove it, and then check if another set exists in the remaining 9 cards.

deck = make_cards()
random.shuffle(deck)
table = deck[:12]
deck = deck[12:]

sets_in_first = []
sets_in_remainder = []

for x in range(100000):
    deck = make_cards()
    random.shuffle(deck)
    table = deck[:12]
    s = find_sets(table)
    if s:
        sets_in_first.append(len(s))
        picked_s = random.choice(s)
        table = [c for c in table if c not in picked_s]
        s = find_sets(table)
        if s:
            sets_in_remainder.append(len(s))


print("Odds a set is still around after the first set is taken: {:.2%}".format(
    sum(sets_in_remainder) / sum(sets_in_first)
))

And the result:

Odds a set is still around after the first set is taken: 28.28%

It appears that if you can spot a set just after your opponent calls "set", there is a ~28% chance that your late-spotted set will still be there after your opponent has collected their three cards (at least, for the very first deal of the game).

This is really interesting. The odds that two non-intersecting sets exist in the first twelve cards is about half the odds that two non-intersecting sets exist in the final 12 cards. That means that, not only should you continue looking for sets as your opponent is collecting cards and dealing new ones, but that that strategy becomes more powerful later in the game (presumably linearly, though I haven't verified).

For more SET fun and other analyses, Peter Norvig has (of course!) done that a great job.

This code was fun to write, and perhaps you can do some more exploration from here!

A Simple Trending Products Recommendation Engine in Python

2017-02-04T00:00:00-08:00

Our product recommendations were boring. I knew that because our customers told us. When surveyed, the #1 thing they wanted from us was better product discovery. And looking at the analytics data, I could see customers clicking through page after page of recommendations, looking for something new to buy. We weren't doing a good job surfacing the back half of our catalog. There was no serendipity.

One common way of increasing exposure to the long tail of products is by simply jittering the results at random. But injecting randomness has two issues: first, you need an awful lot of it to get products deep in the catalog to bubble up, and second, it breaks the framing of the recommendations and makes them less credible in the eyes of your customers.

What do I mean by 'framing'? Let's look at a famous example from Yahoo!

The Britney Spears Effect.

Let's say you're reading about this weekend's upcoming NFL game. Underneath that article are a bunch of additional articles, recommended for you by an algorithm. In the early 2000s, it turned out just about everyone wanted to read about Britney Spears, whether they would admit it or not.

So you get to the bottom of your Super Bowl game preview and it says "You might also like:" and then shows you an article about Britney and K-fed. You feel kind of insulted by the algorithm. Yahoo! thinks I want to read about Britney Spears??

But instead, what if said "Other people who read this article read:". Now...huh...ok - I'll click. The framing gives me permission to click. This stuff matters!

Just like a good catcher can frame a on-the-margin baseball pitch for an umpire, showing product recommendations on a website in the right context puts customers in the right mood to buy or click.

"Recommended for you" -- ugh. So the website thinks it knows me, eh? How about this instead:

"Households like yours frequently buy"

Now I have context. Now I understand. This isn't a retailer shoving products in front of my face, it's a helpful assemblage of products that customers just like me found useful. Chock-full of social proof!

Finding Some Plausible Serendipity

After an awesome brainstorming session with one of our investors, Paul Martino from Bullpen Capital, we came up with the idea of a trending products algorithm. We'll take all of the add-to-cart actions every day, and find products that are trending upwards. Sometimes, of course, this will just reflect the activities of our marketing department (promoting a product in an email, for instance, would cause it to trend), but with proper standardization it should also highlight newness, trending search terms, and other serendipitous reasons a product might be of interest. It's also easier for slower-moving products to make sudden gains in popularity so should get some of those long-tail products to the surface.

Implementing a Trending Products Engine

First, let's get our add-to-cart data. From our database, this is relatively simple; we track the creation time of every cart-product (we call it a 'shipment item') so we can just extract this using SQL. I've taken the last 20 days of cart data so we can see some trends (though really only a few days of data is needed to determine what's trending):

SELECT v.product_id
  , -(CURRENT_DATE - si.created_at::date) "age"
  , COUNT(si.id)
FROM product_variant v
INNER JOIN schedule_shipmentitem si ON si.variant_id = v.id
WHERE si.created_at >= (now() - INTERVAL '20 DAYS')
  AND si.created_at < CURRENT_DATE
GROUP BY 1, 2

I've simplified the above a bit (the production version has some subtleties around active products, paid customers, the circumstances in which the product was added, etc), but the shape of the resulting data is dead simple:

id  age count
14  -20 22
14  -19 158
14  -18 94
14  -17 52
14  -16 56
14  -15 56
14  -14 52
14  -13 100
14  -12 109
14  -11 151
14  -10 124
14  -9  123
14  -8  58
14  -7  64
14  -6  114
14  -5  93
14  -4  112
14  -3  87
14  -2  81
14  -1  19
15  -20 16
...
15  -1  30
16  -20 403
...
16  -1  842

Each row represents the number of cart adds for a particular product on a particular day in the past 20 days. I use 'age' as -20 (20 days ago) to -1 (yesterday) so that, when visualizing the data, it reads left-to-right, past-to-present, intuitively.

Here's sample data for 100 random products from our database. I'm anonymized both the product IDs and the cart-adds in such a way that, when standardized, the results are completely real, but the individual data points don't represent our actual business.

Basic Approach

Before we dive into the code, let's outline the basic approach by visualizing the data. All the code for each intermediate step, and the visualizations, is included and explained later.

Here's the add-to-carts for product 542, from the sample dataset:

The first thing we'll do is add a low-pass filter (a smoothing function) so daily fluctuations are attentuated.

Then we'll standardize the Y-axis, so popular products are comparable with less popular products. Note the change in the Y-axis values.

Last, we'll calculate the slopes of each line segment of the smoothed trend.

Our algorithm will perform these steps (in memory, of course, not visually) for each product in the dataset and then simply return the products with the greatest slope values in the past day, e.g. the max values of the red line at t=-1.

The Code

Let's get into it! You can run all of the code in this post via a Python 2 Jupyter notebook.

Here's the code to produce the first chart (simply visualizing the trend). Just like we built up the charts, we'll build from this code to create the final algorithm.

import matplotlib.pyplot as plt
import pandas as pd
import numpy as np

# Read the data into a Pandas dataframe
df = pd.read_csv('sample-cart-add-data.csv')

# Group by ID & Age
cart_adds = pd.pivot_table(df, values='count', index=['id', 'age'])

ID = 542
trend = np.array(cart_adds[ID])

x = np.arange(-len(trend),0)
plt.plot(x, trend, label="Cart Adds")
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
plt.title(str(ID))
plt.show()

It doesn't get much simpler. I use the pandas pivot_table function to create an index of both product IDs and the 'age' dimension, which just makes it easy to select the data I want later.

Smoothing

Let's write the smoothing function and add it to the chart:

def smooth(series, window_size, window):

    # Generate data points 'outside' of x on either side to ensure
    # the smoothing window can operate everywhere
    ext = np.r_[2 * series[0] - series[window_size-1::-1],
                series,
                2 * series[-1] - series[-1:-window_size:-1]]

    weights = window(window_size)
    weights[0:window_size/2] = np.zeros(window_size/2)
    smoothed = np.convolve(weights / weights.sum(), ext, mode='same')
    return smoothed[window_size:-window_size+1]  # trim away the excess data

smoothed = smooth(
    trend,
    7,
    np.hamming
)
plt.plot(x, smoothed, label="Smoothed")

This function merits an explanation. First, it's taken more-or-less from the SciPy Cookbook, but modified to be...less weird.

The smooth function takes a 'window' of weights, defined in this case by the Hamming Window, and 'moves' it across the original data, weighting adjacent data points according to the window weights.

Numpy provides a bunch of windows (Hamming, Hanning, Blackman, etc.) and you can get a feel for them at the command line:

>>> print np.hamming(7)
[ 0.08  0.31  0.77  1.    0.77  0.31  0.08]

That 'window' will be moved over the data set ('convolved') to create a new, smoothed set of data. This is just a very simple low-pass filter.

Lines 5-7 invert and mirror the first few and last few data points in the original series so that the window can still 'fit', even at the edge data points. This might seem a little odd, since at the end of the day we are only going to care about the final data point to determine our trending products. You might think we'd prefer to use a smoothing function that only examines historical data. But because the interpolation just mirrors the trailing data as it approaches the forward edge, there's ultimately no net effect on the result.

Standardization

We need to compare products that average, for instnace, 10 cart-adds per day to products that average hundreds or thousands. To solve this problem, we standardize the data by dividing by the Interquartile Range (IQR):

def standardize(series):
    iqr = np.percentile(series, 75) - np.percentile(series, 25)
    return (series - np.median(series)) / iqr

smoothed_std = standardize(smoothed)
plt.plot(x, smoothed_std)

I also subtract the median so that the series more-or-less centers around 0, rather than 1. Note that this is standardization not normalization, the difference being that normalization strictly bounds the value in the series between a known range (typically 0 and 1), whereas standardization just puts everything onto the same scale.

There are plenty of ways of standardizing data; this one is plenty robust and easy to implement.

Slopes

Really simple! To find the slope of the smoothed, standardized series at every point, just take a copy of the series, offset it by 1, and subtract. Visually, for some example data:

And in code:

slopes = smoothed_std[1:]-smoothed_std[:-1])
plt.plot(x, slopes)

Boom! That was easy.

Putting it all together

Now we just need to repeat all of that, for every product, and find the products with the max slope value at the most recent time step.

The final implementation is below:

import pandas as pd
import numpy as np
import operator

SMOOTHING_WINDOW_FUNCTION = np.hamming
SMOOTHING_WINDOW_SIZE = 7

def train():
    df = pd.read_csv('sample-cart-add-data.csv')
    df.sort_values(by=['id', 'age'], inplace=True)
    trends = pd.pivot_table(df, values='count', index=['id', 'age'])

    trend_snap = {}

    for i in np.unique(df['id']):
        trend = np.array(trends[i])
        smoothed = smooth(trend, SMOOTHING_WINDOW_SIZE, SMOOTHING_WINDOW_FUNCTION)
        nsmoothed = standardize(smoothed)
        slopes = nsmoothed[1:] - nsmoothed[:-1]
        # I blend in the previous slope as well, to stabalize things a bit and
        # give a boost to things that have been trending for more than 1 day
        if len(slopes) > 1:
            trend_snap[i] = slopes[-1] + slopes[-2] * 0.5
    return sorted(trend_snap.items(), key=operator.itemgetter(1), reverse=True)

def smooth(series, window_size, window):
    ext = np.r_[2 * series[0] - series[window_size-1::-1],
                series,
                2 * series[-1] - series[-1:-window_size:-1]]
    weights = window(window_size)
    smoothed = np.convolve(weights / weights.sum(), ext, mode='same')
    return smoothed[window_size:-window_size+1]


def standardize(series):
    iqr = np.percentile(series, 75) - np.percentile(series, 25)
    return (series - np.median(series)) / iqr


trending = train()
print "Top 5 trending products:"
for i, s in trending[:5]:
    print "Product %s (score: %2.2f)" % (i, s)

And the result:

Top 5 trending products:
Product 103 (score: 1.31)
Product 573 (score: 1.25)
Product 442 (score: 1.01)
Product 753 (score: 0.78)
Product 738 (score: 0.66)

That's the core of the algorithm. It's now in production, performing well against our existing algorithms. We have a few additional pieces we're putting in place to goose the performance further:

Throwing away any results from wildly unpopular products. Otherwise, products that fluctuate around 1-5 cart-adds per day too easily appear in the results just by jumping to 10+ adds for one day.
Weighting products so that a product that jumps from an average of 500 adds/day to 600 adds/day has a chance to trend alongside a product that jumped from 20 to 40.

There is weirdly little material out there about trending algorithms - and it's entirely possible (likely, even) that others have more sophisticated techniques that yield better results.

But for Grove, this hits all the marks: explicable, serendipitous, and gets more clicks than anything other product feed we've put in front of our customers.

How to Buy Expensive Software Part 1: Vendors & Demos

2016-11-20T00:00:00-08:00

Some days it seems like all I'm good for is buying other companies' software.

It's a little ridiculous. I get 5-10 emails a day pitching some SaaS solution. You probably get them too. Recognize any of these subject lines?

Christopher, do you have 15 minutes to connect about my previous note?
Try SaaSly for a week - Get $200
What do Lyft, Hubspot and Instacart all have in common?
wrong person?
Meeting Request - EXCLUSIVE

My blood pressure went up 10% just copy-pasting those. On the other hand, I can't fault these sales folks because, once in a while, I'm actually in market and end up having to buy one of these things.

And that's what this is about -- buying expensive software.

What's "expensive"?

For our purposes, "expensive" is anything that costs enough money that the vendor has dedicated sales people who will actually schedule meetings with you, answer your questions, prepare demos, and negotiate pricing. Others have written about this before, but basically there is some threshold where prices jump from, like, $399/mo to $1500/mo. It's sort of hard to find software that costs $900/mo¹ because I'm not going to spend that much money without demos and talking to a sales rep, but it's not really a large enough sale to support that kind of sales infrastructure either².

For that reason, most real B2B SaaS solutions start somewhere in the $2000/mo range. If you're a growing company, at some point you'll suddenly find that every department in the company wants to buy SaaS products that are each $2000/mo. Yikes.

I'll walk you how these deals come together, what to expect at each stage of the purchasing process, help you ask the right questions, and ultimately negotiate price, while still remaining on great terms with your vendor.

Entering the Market

So let's pretend you're in market for a business intelligence tool. Or a CRM. Or an inventory management solution. Or an email marketing system. Or possibly you're in market for all of these and SalesForce has promised that The Force.com Platform (r)(tm) can do all of this, plus get you out of unpaid parking tickets, and you will have lots of Success and not too much Software and it's going to be a lovely, platformy day.

The first thing you have to do is pick a couple of vendors. Even if you have a strong idea about who will be the best fit, I always like looking at two vendors, if possible. The competitive dynamics of the deal will give you some negotiating leverage later on, and even if the 'other' vendor ends up being weak, it'll give you just that much more conviction about the solution you picked. Anywhere between 2 and 4 vendors is great, and you can always drop unpromising ones at any point through the process.

Got a couple in mind? Great! If you don't have a contact there already, just fill out a lead form online and I promise you will have a sales rep at your doorstep in no time.

Meet You New Best Friend

This is important to remember: the sales rep is on your side. This is someone who is very interested in both getting this deal done (therefore at a price you can afford, and with the features you want) and in making sure the solution is actually a good fit (because all sorts of things can go sideways after purchase, and that has real financial consequences for the rep).

Sales reps are your friends. They may frustrate you with their inability to answer technical questions, or give you a little bit of the runaround on how a particular feature works, but they are on your side. So be on theirs. When it comes time to put the screws to the vendor on pricing, it's your rep who will go to bat with their managers to get your pricing approved. So be friendly!

The First Phone Call

You've reached out and made contact with your sales rep via email. It's looking good and you are now a warm lead. The next step is an initial call.

It's somewhat unlikely you will actually get a demo of the software in this first call. Totally annoying, yes, but normal. Deal with it. Your rep is first going to want to understand your goals and your company, so she can put together a relevant demo later on. You'll definitely be asked to explain your role in the company. Regardless of whether I'm the sole decision-maker or just helping another department make a purchase, I always make sure the rep knows that I am necessary, but not sufficient, for getting this deal done. This puts me in a position where my concerns and questions have to be taken seriously, but also creates the propsect of additional stakeholders, who I can use as 'the bad guy' later in negotiations, and to buy myself time if I get in a jam.

Now, back to that first call: you have some important diligence of your own that you will need to perform.

But first, you will have to battle your way through the Boring Vendor PowerPoint of Irrelevance. After a few minutes of standard "can you see my screen?" GoToMeeting formalities, you'll probably get a slideshow like this:

Slide 1:

GIANT COMPANY LOGO
Lots of little logos of prominent customers
"Trusted by industry leaders"
Depending on the sophistication of the sales org, they may have taken the dramatic step of including your company's logo. +1 vendor points if so.

Slide 2:

LOTS OF STATS ABOUT RETURN ON INVESTMENT and HOW YOUR COMPANY IS GOING TO GET SUPER LASER NINJA POWERS WHEN YOU BUY THIS THING

Slide 3:

List of modules and features. 90% chance the word "proprietary" appears in at least one of the bullet points.

Slide 4:

I'm not really sure, I've generally tuned out by now. There are definitely a few more slides though.

I feel bad zoning out, but unless there's some buying process I'm unaware of where, like, you end up on sales calls based on that fact that you both swiped right on each other's LinkedIn profiles, you've almost certainly read this information on their website. I just never learn much via cookie-cutter PowerPoint presentations, other than slide transitions look the least crappy when screen sharing².

I save my detailed solution and product questions for demos (which come later) and my approach in the initial call is to just kind of 'uh-huh' my way through the slides as quickly as possible so I can get to the real goal of the first meeting:

Vetting the Vendor

You're about to spend a bunch of money, so make sure you understand who you are partnering with. Here are the questions I like to ask in the first call (when relevant), and why I ask them:

How big is your largest customer? How small is your smallest?
- Like Goldilocks, I want to find a vendor that serves customers my size. If I'm a tiny customer, I won't be able to get attention when I need it. And I don't want to be the biggest customer, and the first to run into scaling challenges.
How many customers do you have? How many engineers? Are you profitable?
- How big is this company? Do they serve lots of companies just like mine? Can they fix problems quickly? Are they a startup that might suddenly go out of business?
Where are you located? What's the support model? Is there phone support?
- Especailly important if this is critical line-of-business software! I can deal with my data viz tool going down for an hour, but my warehouses just go idle if my inventory management tool is down. So what I really want to know here is: when the shit hits the fan, can I talk to a live human being who can fix the problem? I've had vendors with great phone support, but their engineers are all in Asia, so all they could offer was sympathy until 10pm when folks started waking up.
What's your target customer? What parts of the market are important to you in the next 2 years?
- It may be the case that you are a pretty good fit for this vendor today, but their roadmap has them focusing down-market on smaller companies in the next 2 years, and the features that are getting built aren't going to be ones you value.
Will I be able to do reference calls with similar customers?
- Many vendors won't actually set up reference calls until much later in the process (I can't blame them -- you can't go out hassling your customers for a call every time a lead asks this question) but it's important to know you'll be able to do it later. If the answer is "We don't do references", that's a major red flag for me.
Give me an estimate: How long will this take to implement?
- I've heard vendors tell me "30 days" but it turns out they are running a 45-day project backlog, so total time to launch is much longer. If there's a big range, you can ask what's typical, and then "What's the fastest you've ever seen a company implement?". You know best whether you're going to be typical, or fast. This question is also a great gut-check for you own knowledge; if you had "a week from purchase to go-live" in your head, and then vendor is saying 90 days is typical, maybe you don't totally understand what you're getting into.
Give me an estimate: How much is this going to cost?
- This will be the first and last time you talk about price until the end of the process. I try to get a sense of first-year total cost of ownership (TCO) by asking about software costs, implementation costs, and asking if there are any other types of costs I should be aware of. I generally say something like: "Based on what you know about what I'm trying to accomplish, what kind of costs do companies like mine generally see in the first year? Giving me a range is fine, and I won't hold you to it. I just want to make sure we're in the same ball park."

There aren't right and wrong answers to the above, but trust your instincts. You really are going to be partners in making this work, at least through the sales and implementation period. If the vendor seems good, but the sales rep seems like an idiot, that's a red flag. If the rep is great, but the product story seems thin, that's a red flag. You're about to spend a lot of money, with a contractual committment that will be hard to unwind. Take your time, be thorough, and build conviction.

Getting the demo

To get the most out of a demo, I prepare a list of requirements and questions in advance, and drill deep into the weeds during the demo. Don't be afraid to take the technical sales person off script. This serves two functions. First, it pressure-tests the system a bit. Is it flexible? Is it intuitive? Are the UX and data metaphors it uses ones that make sense to you?

Second, you can verify that the capabilities advertised actually work the way you need them to work.

I've never seen intentional dishonesty during a demo, but it's easy to interpret a question in a way that allows them to say "yes, we do that", but might not actually work for you. A real example: I was buying an inventory management system and asked "do you support EDI?" (EDI is a standard communication protocal for tracking the status of orders). The vendor said yes, and they do support it, but it turned out only for outbound orders, not inbound, which is where I needed it. You owe it to your company to be tenacious and dig in.

My last tip for demos is to have your screenshot tool of choice configured and ready to rock. These things can move fast so it's great to be able to save screenshots of each part of the demo to review later, and to send to colleagues who might also have questions. If you're getting demos of multiple solutions, this is especially helpful as they tend to run together in your memory after a while.

What's next?

You're honing in on a decision - congratulations! Hopefully you have at least two vendors that look OK (you only need one, but having two is great for negotiation purposes). Next up you'll request a pricing quote, a statement of work (a scope), and a contract. Stay tuned for part 2 - how to negotiate price, and get the deal done.

Footnotes

1: The exceptions are things that start of cheap and get expensive based on usage.

2: I think there may be an opportunity for a start-up that doesn't really innovate on product but instead makes, e.g., a perfectly decent mid-market CRM product and innovates instead on the sales process by somehow avoiding it entirely, and charging, like, $900/mo, which would radically undercut their competitors.

3: The answer is: They all look terrible. Save the special effects budget for something else.

Photography & Journalism with Darcy Padilla & Julian Cox - Opening Remarks

2016-11-19T00:00:00-08:00

On November 17th, I chaired an event at the de Young Museum related to the opening of the exhibit Danny Lyon: Message to the Future. It's there through April 30th, 2017 and I encourage anyone in San Francisco during its run to see it. Danny Lyon is a remarkable photographer and artist, with a long history of nosing into important social themes, at just the right time. The work of Bay Area photographer and journalist Darcy Padilla is closely related. She has spent her career in similar pursuits, with haunting and memorable results.

Through ArtPoint, the young patron's group affiliated with the museums, we hosted a conversation between Darcy and Julian Cox, the curator who put together the Danny Lyon exhibit, in collaboration with the artist, over the past few years. The discussion between Julian and Darcy focused on her work, and the broader concerns of socially conscious photo-journalism. My opening remarks are below.

Remarks

The tangled intimacy of artist and subject is what grabbed me the first time I saw Danny Lyon's work, and Darcy Padilla's.

With Lyon, it was a photo from The Bikeriders. A young man, about 20, with a pock-marked face, leering out at me -- a face strikingly similar to my father's, and wearing a motorcycle club jacket that my dad would have been jealous of, having flirted with, but never quite fully embraced, his own rebellious side.

And with Darcy, it was The Julie Project, the heart-wrenching, decade-and-a-half-long photo-journalism story of Julie, a mother trapped in relentless poverty, as she lived and died with drugs, AIDS, and abuse. I came across The Julie Project at work, which I would not recommend, as my colleagues are still worried about what caused me to sob at my desk in the middle of a work day.

The images from Darcy, an award-winning photojournalist and our guest here tonight, and Danny Lyon, in whose work you will find many parallels, and you'll have the opportunity to view after this program, these images are full to the brim with empathy.

These are images of immediate proximity. Of immediate understanding -- to people and situations otherwise hard to see, on the margins of society.

Darcy & Danny have spent their careers documenting people, stories, and narratives that are simply not in our usual line of sight.

They have both worked extensively in prisons, surfacing the intentionally and knowingly hidden away. And both have documented systemic poverty and disenfranchisement.

It is hard work to see from the perspective of others. It takes effort and exertion and intention. These artists do this work on our behalf so we can transmute from stories that would otherwise simply fade from our collective consciousness.

Tonight, we have a unique opportunity to learn more. Julian Cox, chief curator of the Fine Arts Museums of San Francisco, will speak with Darcy about present day photo-journalism, and her work. Afterwards, please explore the exhibit Danny Lyon: Message to the Future, which Julian has organized, and is the first comprehensive retrospective of Lyon's work. It will remain open until 10pm, in the far galleries on this floor.

On behalf of ArtPoint, and the Fine Arts Museums of San Francisco, we welcome you, our audience, and Julian and Darcy. Thank you, and enjoy.

Actually Understanding Timezones in PostgreSQL

2016-08-12T12:09:00-07:00

Timezones are the worst. We should all just live in UTC, reasonable sunset hours be damned. And learning about timezone implementation details is incredibly boring and esoteric¹, so most developers are not inclined to figure out what the hell is actually going on until they really need to. Like me, yesterday, when my slightly-imprecise understanding of how PostgreSQL handles timezones ended with a nervous breakdown at my desk, and the marketing department looking very concerned.

My query results seemed impossible! Down was up, up was down, Pacific Time suddenly had no meaning, and I imagined a bunch of National Time Service officials in Greenwhich sniggering at me.

But it turns out all is indeed right with the world, everything makes sense, and I totally know what's going on, now. Follow along (you just need the ability to run SQL on PostgreSQL) and join me in the basking glow of timezone comprehension.

If you make it all the way through, I promise you'll know enough to puzzle out any Postgres timezone question you may have - with the exception of my mystery question at the end, which is totally insane and inexplicable.

Easy Sample Data

To start, let's make a test table. Easy.

CREATE TABLE test
(
    ts_naked timestamp without time zone,
    ts_tz timestamp with time zone
);

One tz-unaware timestamp field, and one that knows about timezones. We'll unpack what "knows about timezones" means shortly.

And I'll also assume you have a PostgreSQL server, with a default of UTC time, via the timezone parameter into postgresql.conf being set like this: timezone = 'UTC'.

Insertin' Stuff

Let the fun begin! Let's first insert some data:

INSERT INTO
    test (ts_naked, ts_tz)
VALUES
    (TIMESTAMP '2016-08-12 10:22:31.949271-07',
    TIMESTAMP WITH TIME ZONE '2016-08-12 10:22:31.949271-07',);

Note that both of the inserted values have a -07 at the end, indicating a -7 hour offset from UTC, which, given that I'm writing this in California (normally a -08 offset), in August, means that daylight savings time is in effect, thus changing my local UTC offset by an hour, to UTC-07. Phew. Ok. We're doing great.

But wait a minute! I just inserted an offset time zone into BOTH my 'naked' timestamp field (that doesn't know about timezones) and the tz-aware field. So what happens when I get the data back out?

#=> SELECT * FROM test;
          ts_naked          |             ts_tz
----------------------------+-------------------------------
 2016-08-12 10:22:31.949271 | 2016-08-12 17:22:31.949271+00

Hm, ok. So basically the naked timezone just took the inserted timestamp at face value and wrote it on in. It completely ignored the -07 at the end. And yep, that makes sense. The PostgreSQL docs confirm it:

PostgreSQL never examines the content of a literal string before determining its type, and therefore will treat both of the above as timestamp without time zone. To ensure that a literal is treated as timestamp with time zone, give it the correct explicit type:

It's not enough to provide the -07 at the end, you need to actually say TIMESTAMP WITH TIME ZONE. But also note that postgres did not save the timezone information for the tz-aware field; this is a really important distinction. Postgres TIMESTAMP WITH TIME ZONE fields do not actually store any TZ info, they just expect TZ info to be present when the fields are written to. Under the covers, Postgres converts it to UTC and internally says "yep, great, I am confident that I am storing this timestamp, normalized to UTC". With that confidence, Postgres can now perform any timezone conversion you'd like. But there's no way to ever retrieve the fact that I originally sent down a timestamp with a UTC-07 offset; that information is gone forever.

Readin' Stuff

Right, so we have data, let's get it back out and see what happens.

First, let's deal with our stupid ts_naked field. It's pretty easy to understand. When we wrote to it, it interpreted the timestamp as literal, and basically just burned it straight into the database. So, no surprise, bad shit happens when you try to use it with timezones:

#=> select ts_naked AT TIME ZONE 'America/Los_Angeles' from test;
           timezone
-------------------------------
 2016-08-12 17:22:31.949271+00

#=> select ts_naked AT TIME ZONE 'UTC' from test;
           timezone
-------------------------------
 2016-08-12 10:22:31.949271+00

Yeah...that's pretty wacky. Postgres has no idea wtf time you actually stored, so it just treats it as whatever the current server timezone is (UTC in our case, from setting timezone in the postgresql.conf file), applies the offsets you requested, and kinda throws up its hands and gives up by returning a timestamp without an offset (as indicated by the +00). So that's not very useful. Don't use timestamps without time zones in PostgreSQL. There, easy.

It's also worth noting at this point that Postgres will let you apply AT TIME ZONE conversions to date fields. I think this is nuts and Postgres should at least throw up a warning, as it's very weird to request a date at a specific timezone and get a totally different date. Thursday August 18th is Thursday August 18th in every time zone; but instead Postgres will sneakily cast it to a datetime, assume midnight, do the TZ conversion, and return a datetime. Check it out:

#=> SELECT ('July 1, 2015'::date AT TIME ZONE 'America/Los_Angeles')::date;
  timezone
------------
 2015-06-30

I mean...come on.

Ok, enough with things you shouldn't do, let's focus on the interesting one. It behaves like you'd hope...for now.

#=> select ts_tz AT TIME ZONE 'UTC' from test;
          timezone
----------------------------
 2016-08-12 17:22:31.949271


#=> select ts_tz AT TIME ZONE 'America/Los_Angeles' from test;
          timezone
----------------------------
 2016-08-12 10:22:31.949271

Great. We can select our data, and request it in any time zone we want, and nothing weird happens. Although...there is one surprise. Note that we actually did not get any timezone information back from these queries; there's not offset at the end of the timestamps. Postgres returned a TIMESTAMP WITHOUT TIME ZONE!

By requesting the data in a particular timezone, Postgres says "you know the offset, since you just asked for it, so I don't need to give it you". There's no offset specified at the end of the timestamp. I don't particularly like this; we're returning ambiguous information over the wire that we might want later on. Let's try something else.

Recall that a plain-old query will in fact give us a TZ-aware timestamp, in the server time:

#=> select ts_tz from test;
             ts_tz
-------------------------------
 2016-08-12 17:22:31.949271+00

But check this out!

#=> SET timezone='America/Los_Angeles';
#=> select ts_tz from test;
             ts_tz
-------------------------------
 2016-08-12 10:22:31.949271-07

Whaaaaaa??! It turns out that every Postgres session has its own time zone. Whoa. The 'timezone' setting that we specified in the postgresql.conf file actually just provides a default timezone value to every session, nothing more than that.

And this is where it's easy to get very surprised. For instance, when Django connects to Postgres, it 'ensures timezone' and slaps the application time zone onto the psycopg2 connection so you get back timestamps in the same timezone you put them in (assuming you are inserting local times).

That's clever, but can lead to incredibly confusion if you are trying to compare results from the same database, but via 2 different connections, which are set to two different time zones. You can always check the current session TZ via SELECT current_setting('TIMEZONE'); or show timezone;.

Isn't this fun??

Offsettin' Stuff

For extra confusion, Postgres will happily let you do this:

#=> select ts_tz AT TIME ZONE 'PST' from test;
         timezone
----------------------------
 2016-08-12 09:22:31.949271

Which may not look confusing, but I promise it is because 'PST' is not actually a timezone! And I can prove it. Postgres keeps a list of all the timezones it knows in a view called pg_timezone_names:

#=> select count(*) from pg_timezone_names;
count
-------
 1204

Holy smokes that's a lot of timezones! Over 1200 of these:

#=> select * from pg_timezone_names order by random() limit 10;
         name         | abbrev | utc_offset | is_dst
----------------------+--------+------------+--------
 Kwajalein            | MHT    | 12:00:00   | f
 posix/W-SU           | MSK    | 03:00:00   | f
 posix/Europe/Vaduz   | CEST   | 02:00:00   | t
 posix/Pacific/Niue   | NUT    | -11:00:00  | f
 Asia/Ashkhabad       | TMT    | 05:00:00   | f
 posix/Australia/LHI  | LHST   | 10:30:00   | f
 posix/Pacific/Tarawa | GILT   | 12:00:00   | f
 posix/America/Adak   | HDT    | -09:00:00  | t
 America/Vancouver    | PDT    | -07:00:00  | t
 Asia/Hovd            | HOVST  | 08:00:00   | t

And not a PST among them:

#=> select * from pg_timezone_names where name='PST';
name | abbrev | utc_offset | is_dst
------+--------+------------+--------
(0 rows)

But it is this other thing, called an 'abbreviation':

#=> select * from pg_timezone_names where abbrev='PST';
          name          | abbrev | utc_offset | is_dst
------------------------+--------+------------+--------
 Pacific/Pitcairn       | PST    | -08:00:00  | f
 posix/Pacific/Pitcairn | PST    | -08:00:00  | f
 posix/SystemV/PST8     | PST    | -08:00:00  | f
 SystemV/PST8           | PST    | -08:00:00  | f

So all of those time zones are the same, and can be abbreviated 'PST'. Note that our friend America/Los_Angeles does not appear here because it takes daylight savings time into account, whereas PST does not and is a true UTC-08 offset.

Most of time this doesn't matter, but it can definitely bite you. For instance (in fact, the issue that motivated this entire post), I wrote a bunch of queries that reported on the number of customer membership renewals each month. These queries all did AT TIME ZONE 'PST' conversions. Our membership renewal process runs at 11:00pm, nightly. We're setting up a new BI tool, and I set the session time zone to Pacific/Los_Angeles, not thinking too hard about it. Well, guess what? For seven months out of the year, that time zone and PST differ by an hour. And 11pm is within 1 hour of midnight. So all of these renewals were suddenly and inexplicably shifted forward by one day.

Recap

That's pretty much it! To recap:

Use only TIMESTAMP WITH TIME ZONE fields.
Remember that Postgres doesn't store time zones; it just normalizes tz-aware timestamps to UTC.
Know what time zone your session is using. It might be set deep in your application code, or in a config file, or in an env var. For your own sanity, make sure all connections to a given database use the same session time zone.
Rely on your session's time zone when possible, rather than using AT TIME ZONE operators in your queries.
Don't ever use AT TIME ZONE transformations on date fields. Only on datetime fields, if at all.
Be aware that offsets and time zones are not the same. Most of the time it doesn't matter, until it suddenly does.

If you want to know even more, I do actually recommend section 8.5.3 of the PostgreSQL documentation. As far as database server technical docs goes, it's pretty entertaining. And if that isn't a strong enough endorsement, then maybe I can tempt you with some gleaned trivia: One reason PostgreSQL does not recommend using simple TIME WITH TIME ZONE (e.g. a time with no date component) is because Moscow's MSK offset means UTC+3 in certain years, and UTC+4 in other years.

Oof.

A Final Mystery

I really do feel like I understand how Postgres handles timezones, and it's all pretty sensible once you know how it works. Except for one lingering mystery. Check this out:

#=> select * from pg_timezone_names where utc_offset='-08:00:00' and is_dst=false;
          name          | abbrev | utc_offset | is_dst
------------------------+--------+------------+--------
 Etc/GMT+8              | GMT+8  | -08:00:00  | f
 Pacific/Pitcairn       | PST    | -08:00:00  | f
 posix/Etc/GMT+8        | GMT+8  | -08:00:00  | f
 posix/Pacific/Pitcairn | PST    | -08:00:00  | f
 posix/SystemV/PST8     | PST    | -08:00:00  | f
 SystemV/PST8           | PST    | -08:00:00  | f

Look at those two extra timezones! Look at their names and abbreviations! They are UTC-08 offsets, but named GMT+8?? GMT and UTC are the same thing! Google could literally not be more categorical about this fact:

But apparently Postgres thinks they are opposites. Or something. If anyone can explain this in the comments, I'd really appreciate it. Thanks!

edit Mystery solved! A very careful reading of section 8.5.3 reveals the secret:

Another issue to keep in mind is that in POSIX time zone names, positive offsets are used for locations west of Greenwich. Everywhere else, PostgreSQL follows the ISO-8601 convention that positive timezone offsets are east of Greenwich.

That is unfortunate.

Footnotes

1: Don't believe me? The actual Postgres time zone documentation opens with this confidence-inspiring passage:

Time zones, and time-zone conventions, are influenced by political decisions, not just earth geometry. Time zones around the world became somewhat standardized during the 1900s, but continue to be prone to arbitrary changes

Eight Surprises During My First Year as a Father

2016-07-30T00:00:00-07:00

I have a 13-month-old son named Dashiell (pronounced Dash-uhl, and I'm sorry in advance to schoolage Dashiell, whose teachers will all say "Dah-sheel?" on his first day). He's awesome. I mean...I'm not super objective, but even so, he's definitely a good dude. We're going to keep him.

Most of my friends don't yet have kids, so I've been keeping a mental list of things that no one told me, that surprised me in this first year. With one exception, these eight things are all observations, not advice. Back-seat drivers are annoying, but it's nice to have a heads-up if a large bend in the road is coming.

1. The amount of stuff the baby owns trends towards the amount of stuff you own.

A baby is a tiny human being, but with full, adult-sized equipment needs. As a long-time dog owner, my pre-baby mental model was: Baby is size of dog; therefore, baby needs similar amount of stuff as dog. This is not correct.

The amount of stuff for the baby will, over time, trend towards the amount of stuff any other human being owns. After our baby shower, if you looked around our house, it probably appeared that, in utero, he already owned the majority of the household possessions. Books, toys, furniture, bottles, diapers, burp clothes, bouncers, clothes. Get ready for a massive influx of things.

2. You are not going to answer emails at 3am when the baby is up.

Prior to actually having a kid, I thought that during the first few months of life, when the baby is going to be up in the middle of the night, I would use that time productively; working, answering emails, etc. I'd be tired, but I'd at least get some work done. That is a totally insane idea, but one that I've also heard from my pre-child coworkers.

If you are up in the middle of the night with a baby, it is because you are feeding the baby, or attempting to get the baby back to sleep, or thinking "why are you screaming at me at 3am?". There is no scenario where you are up in the middle of the night, peacefully answering emails while, like, the baby coos on the floor next to you. If you are capable of answering emails, you are capable of going to sleep. You will pick sleep.

3. Development and learning are totally different.

Though I sort of knew children 'developed', it took a newborn baby to fully appreciate the difference between development and learning. A baby is not a human that doesn't know things. It is a human that was, only months prior, a loose ball of barely differentiated cells. You cannot teach a ball of cells how to interact with the world. Nor can you teach much to a newborn.

At each stage, there's a true limit to how much a baby (or child, or teenager, or perhaps adult) is able to learn. Babies need development, not knowledge. Newborn babies are literally unable to look at things. They can't focus their eyes. They are helpless. I've now internalized that, though it slows, development continues throughout life, including into adulthood. The only book I'm aware of that acknowledges this fact is the mind-blowing In Over Our Heads by Robert Kegan. Adults develop too!

4. Sleep and schedule is everything.

Warning: Advice! Get the book Babywise, and do what it says. You child will be sleeping through the night at 12 weeks. Babies that sleep are happy babies. Parents that sleep are happy parents. Babies with happy parents are happy, and parents with happy babies are happy. If this confuses you, then perhaps you should be getting more sleep. But none of this virtuous happiness cycle can happen if everyone is tired, and crabby, and arguing about emptying the dishwasher.

Also, somewhat counterintuitively, setting a good schedule is liberating; if you can predict when your kiddo will be asleep, or hungry, you can plan your day. Buy Babywise. Do it. You are welcome.

5. Everyone does it differently, and that's normal

This article from the New York Times is a great read. It describes the book The Anthropology of Childhood as "the only baby book you'll ever need" because, once you realize that children are reared every which-way, with totally contrary practices, all over the world, and that kids turn out fine regardless, you can chill the heck out and not sweat the details of whether you are "doing it right". Thanks to my handy summary here, you don't even need to read the article. And you definitely don't need to read the book; it's fairly academic and boring. But sort of fascinating at the same time.

6. The maturation process.

To some extent at least, I now believe that "maturation" is simply the process of slowly coming to grips with the fact that you are not the center of the universe. Toddlers, for instance, are not mature.

7. The first six months are a little tough for dads.

It's fascinating to hold your newborn offspring and watch the human development process those first few months. I could stare at this baby forever, etc. But, like, not really. Frankly, it's a litttttttllle bit boring hanging out with an infant. They don't actually do a whole lot. After a few minutes of goo-gooing and tickling I was just...kinda bored. Moms seem to have a bit of a deeper bond those first months. That's OK. It gets a heck of a lot more interesting and fun. My one year old is hilarious and it's a blast hanging out with him.

8. You will feel a little weird. Then you won't.

This time last year: Dash is 4 weeks old. I'm out on a walk with him. He is in his bassinet. It's 6pm on Saturday. I see a bunch of guys vaguely my age headed to the bar, getting ready to go out on a warm(-ish) summer night in San Francisco. They glance at me pushing a stroller. I feel very lame and suddenly much older.

Last weekend: Dash is a little over a year old. It's Saturday evening, we're out for a walk. None of the above registers. I still go out with my friends (though somewhat less frequently), and nothing about me having a kid is weird anymore. I feel like a young parent, and my kid is awesome, and I love my wife. We're almost out of whole milk, so I've picked some up, along with some salmon to grill in the back yard. We run into our neighbor, who has an eight month old, and swap a couple of jokes, and promise, for the 10th time, to get together for dinner soon. Life is good and I am happy.

So that's it.

Basically, just relax. People way less competent that you have raised perfectly reasonable, happy kids. Re-read observation #5 when you get nervous, look forward to awesomely fun times as per point #8, and for the love of god do #4. Good luck and I look forward to seeing our children play against each other in the NBA, but still be civil when mingling backstage before their respective TED talks.

Fig 1. Chubby tiptoes.

Zero Downtime Rebrand

2016-07-12T00:00:00-07:00

Noah Smith and I gave this presentation at the San Francisco Django meetup in March of 2016, shortly after we rebranded ePantry to Grove Collaborative. Here's how we pulled it off.

Welcome folks!

That's us

We rebranded on March 8, 2016. Grove is an aspiration brand that speaks to our mission and to our customers. ePantry is a practical brand based in the how's and what's instead of the 'why'. We're leaving ePantry behind. Concretelly, this means a redesign, renaming everything to Grove, and moving from epantry.com to grove.co (we couldn't get the .com).

Pretty standard stuff. The bulk of the company is run via a Django monolith webservice backend, and a single-page backbone app on the frontend. We also have an iOS app that consumes the same Django webservices.

Here's what the site used to look like under the ePantry brand.

And here's the Grove brand.

We did this really, really fast. Note that we got the new design (colors, fonts, logo) only 2 weeks before launch. Ah, startups.

Cloudflare is awesome. It allowed us to have both epantry.com and grove.co point to the same Heroku app (which can only have a single SSL cert).

Here's how the SSL config works. This is probably a 90% solution, in that if you inspect the certs, you'll see this somewhat janky cloudflaressl cert at the end of the chain, instead of a grove.co cert. We still get the green lock in all browsers.

Foreshadowing! We'll be redirecting everyone from epantry.com to the new grove.co domain. Can you guess what bad thing is going to happen?

This is bad. We are a subscription commerce site. The last thing you want to do is log out every single one of your users. That is not a way to help retention during the changeover. Not to mention our emails are going to be coming from a different domain, with less sender reputation, and customers may not recognize them. Password managers also won't be aware of the new domain.

This is all Noah! He came up with a great way to keep folks logged in (securely) across domains.

It basically operates like an OAuth flow; an anonymous user comes to grove.co, we redirect them to epantry.com, detect that there is an active session, then (on the server) literally copy their session data in the database from the epantry.com session into the new grove.co session, and then finally redirect them to their final destination. The code for this is available on github.

In general (and in addition to the session-exchange middleware) we built middleware to redirect all traffic from the old domain to the new one. However there is some traffic we discovered we should not redirect, and instead wanted served by the requested domain. Spiders can get redirected, but not session-exchanged because they do not reliably store cookies. Webhooks get confused by redirects as well, so we just continue serving any webhooks from the original domain. Eventually, we managed to move all webhooks over to the new domain, but there were enough moving pieces without doing that prior to launch.

This caught us by surprise. With all of the redirects in Cloudflare to enforce https and www, then all of the session exchange middleware, the redirects were adding up fast! Combine that with the crazy redirects that some advertising vendors use to track theie links, and you could be pushing the browser limit of 20 redirects (10 in old versions of IE!) pretty quickly. We didn't end up hitting it in practice, but something to watch out for. Not to mention the performance hit.

Once all of this was set up, we started testing in the wild in production right away. We added a staff check to the middleware so all employees could kick the tires for a couple of weeks on the new domain and try to flush out any issues. Note that the overall redirect is controlled by a Django setting, which maps to an environmental variable. The idea being that when it came time to launch, we could just flip the switch.

We didn't want staff copy-pasting URLs to the new domain accidentally, so we created template tags that make folks very aware, if they were on the new domain prior to launch. Also of note, the Sites framework was totally useless during this entire process since it is designed for totally parallel sites, not a single site living on two domains.

As mentioned on the timeline, we didn't get the design until late in the game. But that didn't mean we couldn't start work! There was a ron of prep to get ready so that, regardless of how the new design turned out, we could get it implemented fast.

The first thing was to clean up the colors. This bash command will find all hex colors in our sass stylesheets so we can see how far we've drifted from a common color palette.

Same thing for font styles. We use Foundation and all typography should be coming from a small handful of framework classes and styling that we override with sass settings. In theory we should just not have any matches to this expression if we're doing it right (hint: we were not).

Ugh.

It's totally amazing how out of control this stuff can get over a few years of rapid development. This was a great opportunity to go back and clean it up. Look at that reduction in colors! And they are all in well-names sass variables for easy resuse. One thing we would strongly encourage for anyone embarking on a project like this is to force their designers to be really consistent with the colors in their mocups. It's easy to say "here's the new palette" but then have subtle differences in different mocked screenshots. That led to a lot of confusion on a short timeline.

Holy moly is font licensing more complex than I anticipated. I'm not really joking when I say "I'm pretty sure you are allowed to read thi." I'm not entirely sure the font license for that title font ("Archer") covers use in a presentation like this. And all of these screenshots are real, from a SINGLE font company, that I had to contend with to purchase and use ONE font. Nuts. If we had known this earlier in the process, I would have made a much stronger recommendation that we just stick to what's available in Google Fonts.

Not to mention all of the additional infrastructure involved in using fonts from multiple sources. Thanks Adobe Type Cloud -- I didn't USED to have to dedicate IT resources to fonts, when we were using Google, but I sure as hell do now.

Go time!

Look at that Git diff! It's bleeding red! That's what you get from cleaning up all that CSS. We deployed all of the server side code weeks in advance of launch. The actual deploy just consisted of a bunch of static assets deploying. We actually did a test deploy at midnight before launch to verify things worked, and then rolled it back. There were probably a handful of very confused night-owl customers. The once piece you can't take back is redirects. Browsers cache them aggressively so if you do have to roll something back, anyone who hit the redirects will still get pointed to your new domain. Redirects are forever.

Here's the real checklist I used for go-live. Each department had their own. It stayed fairly organized thanks to Emacs and Org Mode. A bunch of this stuff could have, in theory, been scripted. But it was easy enough to write out really explicit steps and just do them by hand at launch time. Not everything has to be totally automated.

These charts are from Librato. We piped a bunch of custom metrics in, including number of requests to each domain, and watched for anomalous HTTP codes. You can see the cutover happen in the top chart, around 6am as traffic moves from epantry.com (green) to the new domain (blue). Smooth sailing!

Customers getting redirected to the new domain for the first time got a full-page model popover introducing them to the new brand. We used cookies to remember if they had seen it and/or dismissed it.

The orange line shows our recurring purchases for the month. The green line is our target budget number. If the rebrand went poorly (if customers either didn't like it, or were confused) then the canary in the coal mine would be a drop off in repurchase rate. But we saw no such thing and stayed precisely on track. Success!

The First Thing I Tell New Engineers

2016-06-30T00:00:00-07:00

The first thing I tell junior programmers is: keep going.

If you've been programming a while, you know that feeling of encountering a new project, with a new stack, in a new language. The feeling is: OK. And you roll up your sleeves and get to work.

But do you remember the first time you were in a professional setting, in front of a new project with years of engineer-hours invested in it? You are staring at a foreign IDE for the first time, and a senior engineer has told you to fix a bug. You have 20 new technologies to learn, all at once, and you don't have a solid conceptual model for how they fit together. The syntax might as well be a Dwarf Fortress game, and you're 10 steps away from even being able to figure out what to Google.

It's overwhelming.

And, you know what? I still find it overwhelming. I recently started learning ClojureScript. The recommended first step is basically, "spend 2 years learning emacs". Then learn the Java ecosystem. Then learn Clojure. Then learn ClojureScript. Yikes.

But the difference between wet-behind-the-ears, just-graduated Chris Clark, and the me of today, is that I've had this feeling dozens, or hundreds, of times before. I just barrel through. I know that all of this stuff was built by other engineers. Some of them were smarter than me, some of them weren't. It doesn't matter; I have complete confidence that if I perservere, I will figure it out. No doubts at all.

So that's what I tell new engineers: When you worry you might not figure it out? That you weren't cut out for this? I've been there, and then been there again. But you'll figure it out, I promise. Just keep going.

The Second Thing I Tell New Engineers

The second thing I tell new engineers (specifically, new grads) is:

In college, a tough professor might give you a huge assignment that might take you 20 hours. What a jerk! In the full-time-work world, that's a project that's done by Thursday.

Software at Companies that Don't Sell Software

2016-06-29T00:00:00-07:00

If you're a software professional, there are two types of companies you might work for; those whose products are fundamentally software (they make money directly from technology), and those that make their money some other way, but rely on software. Let's call the first Type A, and the second Type B¹.

Type A Companies

These folks sell software, or otherwise directly monetize their technology.

Microsoft
Twilio
Dropbox

Type B Companies

These guys are enabled by technology, but don't make money from it directly.

FedEx (fundamentally a logistics company)
WalMart (big website, complex internal systems...but a retailer)
Wells Fargo (and other financial institutions)

Biased

Early in my career, one of the things I looked for on resumes was experience at a Type A company. I believed that the best engineers would gravitate away from Type B companies. I thought Type A employees were more likely to be at the top of the talent heap, and more likely to be current with the tools and practices of the trade.

But in the intervening years, two things have become apparent:

Where there is a big, successful business, there are great people², and great systems³.
Engineers with Type B experience are more likely to play well with others.

I can't stand engineering groups with superior attitudes, that aren't supportive of the rest of the organization. A little competition between departments is healthy, but if we're all building bicycles for the mind, then fundamentally our job is to augment others. It's a service job -- we build tools. A humble, cooperative attitude is invaluable.

There are a few reasons I think Type B companies do a better job fostering this attitude.

Software Engineer at a Type A company

Your job is to build awesome technology that customers will pay for.

You are close to the money

It's straightforward to be a technologist at a Type A company. The software you create is very closely tied to revenue, which is ultimately what drives business. The closer you are to the money, the closer you are to the beating heart of a company.

On the income statement, engineering & R&D are a cost centers⁴, but they are also the golden goose that made the business possible in the first place.

In the words of one engineer I worked with "If it doesn't come out of these fingers, it doesn't happen."

Software Engineer at a Type B company

Your job is to support the business goals of the organization.

You are further from the money

In a Type B company, you're a true cost center, and perhaps viewed with a bit of skepticism. Is all this software really necessary?

Your work is more likely to be internal, rather than customer-facing.
It's more likely to be an auxiliary function, as opposed to a market differentiator
- Even at tech-heavy Type B companies this is true^{[3](#footnote3}; Amazon's differentiator is not the quality of their website. It's their selection, pricing power, and market position.
When it's not a differentiator, every project is subject to a build-or-buy decision.

Money is the central pillar of business, and the further from it you are, the less influence you have.

Closer to the Customer

But working for a Type B company can also be a heck of a lot of fun. Your customers are often the folks sitting down the hall from you. Product feedback is instant. You might get a literal pat on the back for a job well done. Your users have less variance in terms of training and environment, and everyone shares a lot of tribal knowledge.

Good engineering is 20% building product right, and 80% building right product. With the tight feedback loop you get developing internally, you're much more likely to get it right. And with a forgiving customer base, often running on the same infrastructure, you can iterate quickly.

A Healthier Mind-set

If I never overhear the following at my company, we're doing good work:

"I can't get cycles from the engineering group."
"Engineering doesn't think it's a priority."
"I'm nervous to ask because I don't want to distract anyone on the engineering team."

Whether engineering is at the top or the bottom of the cultural hierarchy, technology exists to improve the lives of its users. Engineering is a functional group like marketing, or sales, or operations. These teams need to work alongside each other as peers. The cultures in Type B companies are more likely to foster that mind set.

Notes

1: This doesn't strictly apply to only software companies. Boeing, for instance, clearly makes money directly from their technology products...which are not strictly software products. I would describe them more as giant flying tubes of metal.

2: I do still believe that bad engineers are more likely to be hiding out at Type B companies, but that has no bearing if you are trying to recruit top performers. You'll spot them right away. The difference between MIT and Virginia Polytech is not at the top of the class; both have some of the world's best. But at the bottom of the class, there's more room to hide in a larger, less focused organization.

3: Popular, influential open-source projects came out of Type B companies all the time. The New York Times alone produces incredible open source projects, from D3 to their Objective-C Style Guide, to Gizmo. And don't forget that Django was created by the Lawrence Journal-World newspaper.

4: Excluding Amazon Web Services, of course. I'm talking about Amazon the retailer.

Trello vs. Asana

2016-06-23T00:00:00-07:00

In two totally different companies over the past 4 years, across different personalities, teams, and processes, I've seen Asana rejected no fewer than a dozen times in favor of Trello, in spite of multiple executive mandates to use Asana.

In both companies, management (of which I am a part) made a hard push to use Asana. Everyone gave it a shot, and then somehow ended up back on Trello.

I gave up on foisting Asana (or any other task management tool) on my teams in 2013, but I can't quite convince everyone else that it's a futile practice. I think all managers eventually learn this lesson though; it's hard to make teams use tools that don't fit their hands.

But that's something to explore another time. Here, I just want to think about why specifically Asana keeps getting surreptitiously replaced with Trello as soon as management stops paying attention.

Both tools aim to do about the same thing, and have comparable feature sets. They can both operate as Trojan horses; if a department wants to use it, they just start using it; there's no installation or setup period. But management seems to prefer one, and actual workers seem to prefer the other.

Here's what I think is going on:

Asana

There are projects, tags, sections, and subtasks. And they are all kinda-sorta similar and you can pivot tasks along any of those dimensions. On top of all of that, there are roadmaps, and pipelines. You can sort and filter and drill down and through and around on everything.

The result is great for managers. It's easy to see who is working on what, and how tasks are connected, and where things stand.

The result is terrible for employees. What am I supposed to be working on? Where are all my tasks? What is the top priority? Every view gives me a slightly different answer to these questions. I have a general sense that things are slipping through the cracks.

Trello

There are boards. Boards have lists. Lists have cards. Cards have a bunch of stuff. Pretty much the only way to look at anything is via a board, full of lists, full of cards.

This is extremely annoying if you are trying to figure out, for instance, "what is Chris working on right now?", or even "What did Chris do last week?". Because things are spread across boards¹, and because Trello cards don't have any status, there's just no real way to do any kind of reporting.

On the other hand, one of our most used Trello boards is called "Jordan can you please..." and just contains a ton of projects and tasks (list and cards) Jordan is working on. This works extremely well, and it has a funny name, which is always a bonus. If you want Jordan's help with something, guess where you put the card? And if you want to see the status of your card, guess where you look?

In Conclusion

I've had very few meetings to discuss across teams how we might organize our Trello boards. There aren't a ton of options, and who cares anyway, since there's no benefit to standardization since there's no reporting.

I've had a lot of "How should we all use Asana?" meetings.

Trello makes it easy to manage my own work, and that of my immediate colleagues. Asana makes it easy to manage other people's work. No surprise that managers like Asana, but every time they look at the screens around the office, everyone is using Trello.

Notes

1: Not to mention I still cannot for the life of me figure out how organizations or teams really work in Trello.

Pushing Bits with Python

2016-06-17T00:00:00-07:00

When was the last time you needed to directly manipulate a bunch of binary data? Most engineers run across it only in the context of bit flags and masks in C++ or Java.

It's a little rarer to muck about with direct binary file data. It's nonetheless quite a bit of fun, and a good party trick¹. And, how are you going to solve the next Cicada 3301 puzzle without being able to XOR together a bunch of Tor Onion addresses with mysterious audio signals²?

If you want to learn, then good news -- Python has you covered! The awesome bitstring module makes it simple to take raw data in whatever form you have it (bits, hex, octal, a file stream) and inspect or manipulate it. Be sure to pip install bitstring to follow along. Away we go!

Create a BitArray from scratch

To start, let's use bitstring to create a raw 12-bit binary object, and initialize it to 256. Then we'll take a look at the object in a bunch of different raw formats.

from bitstring import *

twofiftysix = BitArray(uint=256, length=12)

Note that we specified the length of the BitArray so that we have enough bits to convert to Hex and Octal. If we just took 256 as 100000000, then we can't display a corresponding Hex string because each Hex character represents 4 bits, and 256 in binary is only 9 bits long. By specifying a length of 12, our BitArray is padded with 3 leading 0s, and we can convert cleanly into Hex and Octal.

Show me the money!

print "Int: %s, Binary: %s, Hex: %s, Oct: %s" %
    (twofiftysix.uint, twofiftysix.bin, twofiftysix.hex, twofiftysix.oct)

> 'Int: 256, Binary: 000100000000, Hex: 100, Oct: 0400'

Manipulation

We can manipulate the bits directly as well. Here we'll create a couple of BitArrays, and XOR them together. Bitstring overrides the ^ operator to perform an XOR.

bits1 = BitArray(hex='2ba49fe')
bits2 = BitArray(hex='f55e513')

print bits1.bin
print bits2.bin
print "                    XOR'ed ="
print (bits1 ^ bits2).bin
print "\nAnd in hex:"
print (bits1 ^ bits2).bin + " = '" + (bits1 ^ bits2).hex + "'"

Results:

0010101110100100100111111110
1111010101011110010100010011
                    XOR'ed =
1101111011111010110011101101

In hex:
1101111011111010110011101101 = 'defaced'

Did you know 'defaced' is the longest english word you can spell with just ABCDEF! Another awesome fact for your cocktail party banter.

File type detection

Here's something a little more practical. Let's say we have 4 images, with no file extensions, called mystery1, mystery2, mystery3, and mystery4. Some quick Googling will tell us the leading bits of some common image types. We can load up the images, inspect the first bits, and figure out the file type.

types = {
    'bmp': '424d',  # Converted to ASCII, this is 'BM'
    'gif': '474946',  # Converted to ASCII, this is 'GIF'
    'jpeg': 'ffd8ff',
    'png': "89504e470d0a1a0a"
}

images = [
    'mystery1',
    'mystery2',
    'mystery3',
    'mystery4'
]

def detect(name, bits):
    for img_type, pattern in types.items():
        try:
            if bits.hex.index(pattern) == 0:
                print "%s is %s (first bits were %s)" % (name, img_type, bits.hex[:8])
                return
        except:
            continue
    print "Could not identify %s. First bits: %s" % (name, bits.hex[:4])

for i in images:
    with open(i, 'r') as f:
        img = Bits(f)
        detect(i, img)

Problem solved!

mystery1 is jpeg (first bits were ffd8ffe1)
mystery2 is gif (first bits were 47494638)
mystery3 is png (first bits were 89504e47)
mystery4 is bmp (first bits were 424d0072)

Converting to & from ASCII

One last trick; using the built-in binascii library, we can convert ascii to bits and back.

# The built-in binascii library can convert binary to bits
hex = binascii.hexlify('foobar')
bits = BitArray(hex=hex)
print bits.bin

We get:

011001100110111101101111011000100110000101110010

...which we can then convert back to an ASCII string:

print binascii.unhexlify(bits.hex)
> foobar

Notes

1: If this does not impress, you are obviously going to the wrong sorts of parties.

2: For what it's worth, this is literally the reason the learned how to do this in the first place. Is that embarassing, or cool? I guess it depends what kind of party you're at.

A Simple Content-Based Recommendation Engine in Python

2016-06-09T05:15:00-07:00

Let's pretend we need to build a recommendation engine for an eCommerce web site.

There are basically two approaches you can take: content-based and collaborative-filtering. We'll look at some pros and cons of each approach, and then we'll dig into a simple implementation (ready for deployment on Heroku!) of a content-based engine.

For a sneak peak at the results of this approach, take a look at how we use a nearly-identical recommendation engine in production at Grove.

How Content-Based Recommenders Works

Content-based systems are the ones that your friends and colleagues all assume you are building; using actual item properties like description, title, price, etc. If you had never thought about recommendation systems before, and someone put a gun to your head, Swordfish-style, and forced you to describe one out loud in 30 seconds, you would probably describe a content-based system. "Uhh, uhh, I'd like, show a bunch of products from the same manufacturer that have a similar description."

You're using the actual attributes of the item itself to recommend similar products. This makes a ton of sense, as it's how we actually shop in the real world. We go into the Toaster-Oven aisle and look at all the toaster ovens, which are probably physically arranged on the shelf according to brand, or price, or ability to also cook a full turkey in under 30 minutes.

Where Content-Based Falls Short

On most eCommerce sites it's already easy enough for folks to browse the toaster oven category. What we really want is a recommendation system that drives incremental sales (e.g. sales that would not have happened otherwise). If a customer is looking at the product details page for Harry Potter and the Chamber of Secrets, and your recommender shows Prisoner of Azkaban, and the customer buys it, the data scientists back at Random House HQ should not be high-fiving. It's a safe bet that that customer already knew there were more than two books in the series and would have bought Prisoner of Azkaban anyway. It was not an incremental sale.

How Collaborative Filtering Recommenders Work

We need another approach. Enter Collaborative Filtering, or CF. The big idea behind CF is also pretty intuitive; the product someone is most likely to buy, is the product that a bunch of people like you also bought. Sure, this can lead to the Harry Potter situation, but it's much better at making recommendations from further afield, from deeper in the product catalog. It's more robust against problems like typos ("Harry Pooter" still gets recommended), and when measured in the real world in terms of generating incremental sales, generally beats the pants off pure content-based systems.

While the big idea behind CF is intuitive, there is one aspect that you will definitely have to explain to colleagues many, many times. Pure CF systems have no knowledge whatsoever about the products they are recommending! To the system, it's just a giant grid of product IDs and user IDs, representing who bought what. It's deeply counterintuitive that CF algorithms often see no measurable performance improvements when they are hybridized with content-based systems. Surely knowing something about the items you are recommending must help a little, right?

Nope.

In most cases, essentially 100% of the 'signal' is retrievable from a simple matrix of who bought what.

So why on earth would you use content-based approaches?

When Content-Based Approaches Make Sense

But sometimes CF isn't a viable option; let's say we want to make recommendations to a customer viewing a product details page, who just got there from a Google SERP link. We don't know anything about this customer, so we can't build a matrix of purchases. But we can use a content-based system to recommend similar products. In that sense, content-based recommenders can solve the "cold-start" problem that CF systems have.

They can also provide a measure of automated curation when you have a strong indication of buying intent for a particular product (like when a lead comes from a Google search for a term related to that product). If you're interested in the Nike Pro Hypercool Fitted Men's Compression Shirt, you might also love the Nike Pro Hypercool Printed Men's Tights. A content-based engine rocks at picking up related products like this, without a bunch of manual curation (those products don't appear together in the 'pants' category, nor in 'shirts').

Let's Build it with TF-IDF.

Like many algorithms, we can use a bunch of off-the-shelf libraries to make life pretty easy. As I walk through the approach, bear in mind that the entire implementation is going to ultimately be fewer than 10 lines of Python. But before we get to the big reveal and look at the code, let's talk through the approach.

I've provided a sample dataset of outdoor clothing and products from Patagonia. The data looks like this, and you can view the whole thing (~550kb) on Github.

| id | description                                                                 |
|----|-----------------------------------------------------------------------------|
|  1 | Active classic boxers - There's a reason why our boxers are a cult favori...|
|  2 | Active sport boxer briefs - Skinning up Glory requires enough movement wi...|
|  3 | Active sport briefs - These superbreathable no-fly briefs are the minimal...|
|  4 | Alpine guide pants - Skin in, climb ice, switch to rock, traverse a knife...|
|  5 | Alpine wind jkt - On high ridges, steep ice and anything alpine, this jac...|
|  6 | Ascensionist jkt - Our most technical soft shell for full-on mountain pur...|
|  7 | Atom - A multitasker's cloud nine, the Atom plays the part of courier bag...|
|  8 | Print banded betina btm - Our fullest coverage bottoms, the Betina fits h...|
|  9 | Baby micro d-luxe cardigan - Micro D-Luxe is a heavenly soft fabric with ...|
| 10 | Baby sun bucket hat - This hat goes on when the sun rises above the horiz...|

That's it; just IDs and text about the product in the form Title - Description. We're going to use a simple Natural Language Processing technique called TF-IDF (Term Frequency - Inverse Document Frequency) to parse through the descriptions, identify distinct phrases in each item's description, and then find 'similar' products based on those phrases.

TF-IDF works by looking at all (in our case) one, two, and three-word phrases (uni-, bi-, and tri-grams to NLP folks) that appear multiple times in a description (the "term frequency") and divides them by the number of times those same phrases appear in all product descriptions. So terms that are 'more distinct' to a particular product ("Micro D-luxe" in item 9, above) get a higher score, and terms that appear often, but also appear often in other products ("soft fabric", also in item 9) get a lower score.

Once we have the TF-IDF terms and scores for each product, we'll use a measurement called cosine similarity to identify which products are 'closest' to each other.

Luckily, like most algorithms, we don't have to reinvent the wheel; there are ready-made libraries that will do the heavy lifting for us. In this case, Python's SciKit Learn has both a TF-IDF and cosine similarity implementation. I've put the whole thing together in a Flask app that will actually serve recommendations over a REST API, as you might do in production (in fact, the code is not very different from what we actually do run in production at Grove).

The engine has a .train() method that runs TF-IDF across the input products file, computes similar items for every item in the set, and stores those items along with their cosine similarity, in Redis. The .predict method just takes an item ID and returns the precomputed similarities from Redis. Dead simple!

The engine code in its entirety is below. The comments explain how the code works, and you can explore the complete Flask app on Github.

import pandas as pd
import time
import redis
from flask import current_app
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import linear_kernel


def info(msg):
    current_app.logger.info(msg)


class ContentEngine(object):

    SIMKEY = 'p:smlr:%s'

    def __init__(self):
        self._r = redis.StrictRedis.from_url(current_app.config['REDIS_URL'])

    def train(self, data_source):
        start = time.time()
        ds = pd.read_csv(data_source)
        info("Training data ingested in %s seconds." % (time.time() - start))

        # Flush the stale training data from redis
        self._r.flushdb()

        start = time.time()
        self._train(ds)
        info("Engine trained in %s seconds." % (time.time() - start))

    def _train(self, ds):
        """
        Train the engine.

        Create a TF-IDF matrix of unigrams, bigrams, and trigrams
        for each product. The 'stop_words' param tells the TF-IDF
        module to ignore common english words like 'the', etc.

        Then we compute similarity between all products using
        SciKit Leanr's linear_kernel (which in this case is
        equivalent to cosine similarity).

        Iterate through each item's similar items and store the
        100 most-similar. Stops at 100 because well...  how many
        similar products do you really need to show?

        Similarities and their scores are stored in redis as a
        Sorted Set, with one set for each item.

        :param ds: A pandas dataset containing two fields: description & id
        :return: Nothin!
        """

        tf = TfidfVectorizer(analyzer='word',
                             ngram_range=(1, 3),
                             min_df=0,
                             stop_words='english')
        tfidf_matrix = tf.fit_transform(ds['description'])

        cosine_similarities = linear_kernel(tfidf_matrix, tfidf_matrix)

        for idx, row in ds.iterrows():
            similar_indices = cosine_similarities[idx].argsort()[:-100:-1]
            similar_items = [(cosine_similarities[idx][i], ds['id'][i])
                             for i in similar_indices]

            # First item is the item itself, so remove it.
            # This 'sum' is turns a list of tuples into a single tuple:
            # [(1,2), (3,4)] -> (1,2,3,4)
            flattened = sum(similar_items[1:], ())
            self._r.zadd(self.SIMKEY % row['id'], *flattened)

    def predict(self, item_id, num):
        """
        Couldn't be simpler! Just retrieves the similar items and
        their 'score' from redis.

        :param item_id: string
        :param num: number of similar items to return
        :return: A list of lists like: [["19", 0.2203],
        ["494", 0.1693], ...]. The first item in each sub-list is
        the item ID and the second is the similarity score. Sorted
        by similarity score, descending.
        """

        return self._r.zrange(self.SIMKEY % item_id,
                              0,
                              num-1,
                              withscores=True,
                              desc=True)

content_engine = ContentEngine()

Run It Yourself!

If you actually want to try this out, it's very easy. Follow the instructions in the readme and you'll have it running locally in no time, using the sample Patagonia data. The engine is ready to be deployed to Heroku as well.

Take that, collaborative filtering! #contentbased

How Celery Chord Synchronization Works

2015-03-26T10:20:00-07:00

Celery is a powerful tool for managing asynchronous tasks in Python. The basic model is synchronous Python code pushes a task (in the form of a serialized message) into a message queue (the Celery "broker", which can be a variety of technologies - Redis, RabbitMQ, Memcached, or even a database), and worker processes pull tasks off the queue and execute them. But Celery's 16,000 lines of application code certainly provide a lot more functionality than a simple task queue. Celery exposes a number of powerful synchronization (or "workflow" in Celery parlance) primitives - ways to execute groups of tasks together, chain async task results in a synchronous manner, or execute a callback after a group of tasks have finished executing.

At Grove, we make extensive use of this final primitive, called a "chord" in Celery. Here's a trivial chord example, from the Celery docs:

>>> from celery import chord
>>> from tasks import add, tsum

>>> chord(add.s(i, i)
...       for i in xrange(100))(tsum.s()).get()
9900

This queues up 100 'add' tasks, the results of which Celery aggregates into a list, which is passed into the callback function (tsum) when all of the tasks have finished executing.

We use chords for all sorts of tasks. For example, every night we charge recurring shipments, and send out an internal email with the results of how many charges succeeded, and how many failed due to things like expired credit cards. The charge tasks are all asynchronous, distributed across many workers, and the final email is the chord's callback function.

Pretty cool! But wait...how exactly does this coordination work? The Celery docs leave this somewhat cryptic comment:

"The synchronization step is costly, so you should avoid using chords as much as possible. "

Err...

Reading a bit further, we learn that:

By default the synchronization step is implemented by having a recurring task poll the completion of the group every second, calling the signature when ready.

That does sound a bit costly. But wait! There's more!

This is used by all result backends except Redis and Memcached, which increment a counter after each task in the header, then applying the callback when the counter exceeds the number of tasks in the set. Ah-ha! The most common broker backends for Celery are Redis and RabbitMQ, which will serve as exemplars for the two types of synchronization as we dig in.

Redis is often described as a "swiss army knife" for data. It functions great as a message broker, but also as a general-purpose (non-relational) data store. This means that Celery can store a count in Redis, of the number of tasks originally in the chord, and increment that counter every time a task completes. Here's the relevant code from the Celery source:

_, readycount, totaldiff, _, _ = client.pipeline()               \
     .rpush(jkey, self.encode([1, tid, state, result]))          \
     .llen(jkey)                                                 \
     .get(tkey)                                                  \
     .expire(jkey, 86400)                                        \
     .expire(tkey, 86400)                                        \
     .execute()

     totaldiff = int(totaldiff or 0)

     try:
         callback = maybe_signature(request.chord, app=app)
         total = callback['chord_size'] + totaldiff
         if readycount == total:
             decode, unpack = self.decode, self._unpack_chord_result
                ...

This code (which is part of a larger function) executes every time one of the chord's subtasks completes by binding to the on_chord_part_return property. Through some clever Redis pipelining, the count gets incremented, then retrieved into readycount. Then, if readycount==total, the chord callback gets executed. Cool!

Is it expensive, as the docs claim? Well...sort of. It causes a slew of Redis commands to fire after every subtasks completes, but it uses connection pooling, and the pipeline redis primitive, so there is only one round-trip to the broker. It depends on your use case whether you consider this expensive or not. it seems like a small price to pay for the synchronization.

Now that we understand Redis, let's see how RabbitMQ does it.

RabbitMQ is a message queue. It is not designed to persist arbitrary data, but is purpose built as a broker. There isn't a place to store a counter, so Celery relies on a polling strategy to determine if the chord is ready to complete. The code for this is spread out across a number of source files, but it all starts with apply_chord() in the base broker class (which RabbitMQ, referred to as 'amqp' in Celery, inherits from).

When the chord starts, apply_chord calls callback_chord_unlock, which in turn queues up the built-in celery.chord_unlock task. Here's the crucial bit of code in chord_unlock that polls for the completion of the subtasks:

try:
    ready = deps.ready()
except Exception as exc:
    raise self.retry(
               exc=exc, countdown=interval, max_retries=max_retries,
               )
...
callback.delay(ret)

If the subtasks are ready, the callback gets queued for execution. Otherwise, chord_unlock gets queued for a retry. So...is this expensive? Well, maybe. Certainly more so than checking a counter. Check out how ResultSet.ready() works -- it checks the .ready() property of each subtask, which ultimately results in a call to _get_task_meta for each subtask. Celery does a good job caching the task metadata, but nonetheless this means examining the metadata for each task, each time the chord_unlock polling task runs. I suspect this is what the docs refer to when they warn of the synchronization being potentially expensive.

So that's it -- an under-the-covers look at how Celery actually coordinates tasks. I was surprised to learn that the strategy varied so radically based on the broker backend, but certainly reaffirms my love for Redis and all its flexibility.

Refactoring for Testability: A Real World Example in Python/Django

2014-04-21T22:25:00-07:00

At ePantry, we strive to have enough automated test coverage that we can deploy with confidence, without being dogmatic about our approach. We certainly aren't a TDD shop, and we don't have rules about code coverage metrics ("no commit can reduce code coverage!"), but it's very unusual that a non-cosmetic change makes it to production without some sort of test coverage.

But it wasn't a straightforward journey - our team didn't have much experience with "good" testing practices. In a few previous companies, tests if they were required at all, were viewed as a burden; "extra work" to tack on to your commit at the very end.

So building a culture around automated testing took time. To the extent we've succeeded, we did it by brute force: we just wrote the damn tests. After a couple of months of this and one by one, we built up enough actually internalize two key lessons:

It's actually faster to develop if you write the tests alongside your code. Clicking around in the browser reproducing bugs is slow and unreliable and not very fun.
Writing tests actually catches a lot of bugs before they make it into production. I'm now physically uncomfortable deploying untested code.

But it took a bit of wandering through the testing wilderness to get there.

The other day at ePantry we refactored a bit of code for testability that I think provides a nice example of both the mechanics and the value of automated testing - something that I think can be tricky to understand from toy examples. The refactoring was simple, but yielded code that is easier to maintain, more composable, and (most importantly) allowed us to deploy a major feature with a high degree of confidence.

First, some background:

At ePantry we generate suggested shipments of household goods based on predicted consumption habits of the household. When a customers signs up, we generate roughly a year of suggested shipments. For instance, here's what a schedule looks like to a user on the front end:

Over time, the household "consumes" those shipments and we need to generate more to keep the calendar populated with future shipments. We wrote a function that would run as an overnight chron job to "top up" the shipments of all of our customers so each customer always has at least a year of shipments on the calendar.

Below is a code snippet as it first appeared in our codebase, sans-tests. The code itself is simple, but it was a bit scary to deploy into production because it runs asynchronously, touches a lot of customers, and would not be easy to unwind if it ran amok. I've removed some bits of logging for clarity, but this is otherwise just as it appeared in the original commit.

def create_shipments_async():
    search_date = add_months(datetime.date.today(), \
                             settings.SHIPMENTS_UNTIL_AT_LEAST_MONTHS)
    customers = Customer.objects \
                 .annotate(last_shipment_date=Max('pantry__shipments__arrival_date')) \
                    .filter(last_shipment_date__lt=search_date) \
                    .filter(card_on_file=True)

    for customer in customers:
        cur_date = customer.last_shipment_arrival_date()
        while cur_date < search_date:
            shipment = customer.pantry.create_next_shipment()
            cur_date = shipment.arrival_date

Here's what it does:

First we find the date in the future that we want all customers to have shipments until. This is set to 12 months in our settings file.
Next we get all of the active customers (those who have a credit card on file) whose last shipment arrives before the cutoff date. These customers need more shipments generated.
Finally, for each of the customers, we loop through and create shipments for each customer until the arrival date of the last shipment is past our target date.

Simple! But also untested and therefore scary. We tested it locally by cloning the production database and running the function through the Django shell. It seemed to work. But that's not good enough. This is too important and too complicated to be left to manual testing. So let's write some unit tests.

As is, this code is difficult to test. It's one big function doing a bunch of stuff and we can't test the components in isolation. If you look at the english language explanation of the code above, there are really three distinct steps, each of which can be separately verified. We can refactor to represent each of those three steps:

def _shipments_until_date():
    return add_months(datetime.date.today(), \
                      settings.SHIPMENTS_UNTIL_AT_LEAST_MONTHS)

def _get_customers_without_enough_shipments(search_date):
    return Customer.objects \
        .annotate(last_shipment_date=Max('pantry__shipments__arrival_date')) \
        .filter(last_shipment_date__lt=search_date) \
        .filter(card_on_file=True)

def _create_until(customer, last_shipment_date, search_date):
    if last_shipment_date < search_date:
        _create_until(customer, \
                      customer.pantry.create_next_shipment().arrival_date, \
                      search_date)

Now what does our top-level function look like?

def create_shipments_async():
    map(lambda customer: _create_until(customer, \
                                   customer.last_shipment_arrival_date(), \
                                   _shipments_until_date()), \
           , _get_customers_without_enough_shipments(_shipments_until_date()))

Piece of cake! Literally one line of code. We map our _create_until function onto the collection of customers without enough shipments. All the detail is in our three component functions (note that _create_until is just a recursive loop - it could certainly be implemented in a more imperative style of that's your thing).

We just need a handful of tests on these functions, and one good test on the high level function, and we gain a tremendous amount of confidence and robustness in the code. I glossed over it here, but I also discovered a few small bugs in our Django ORM query while writing the tests. Here are a few example tests, using Django's test framework and Factory Boy to create mock objects.

from django.test import TestCase
from pypantry.tests.factories import PantryFactory
from schedule.tests.factories import ShipmentFactory
from pypantry.tasks import *
from datetime import timedelta, datetime
from utils.utils import add_months


class TestShipmentGenTask(TestCase):

    def setUp(self):
        self.p = PantryFactory()
        self.c = self.p.customer
        self.c.set_card_on_file('foo')
        self.c.save()
        self.s1 = ShipmentFactory(pantry=self.c.pantry)

    def test_finds_customers(self):
        self.s1.arrival_date = _shipments_until_date() - timedelta(1)
        self.s1.save()
        res = _get_customers_without_enough_shipments(_shipments_until_date())
        self.assertEqual(len(res), 1)

    def test_ignores_customers_with_enough_shipments(self):
        self.s1.arrival_date = _shipments_until_date()
        self.s1.save()
        res = _get_customers_without_enough_shipments(_shipments_until_date())
        self.assertEqual(len(res), 0)

    def test_shipments_get_created(self):
        self.s1.arrival_date = datetime.strptime('15042014', "%d%m%Y").date()
        self.s1.save()
        _create_until(self.c, self.s1.arrival_date, add_months(self.s1.arrival_date, 12))
        # one extra because it goes one shipment "past" the target date
        self.assertEqual(self.p.shipments.count(), 13)

    def test_task(self):
        self.s1.arrival_date = _shipments_until_date() - timedelta(1)
        self.s1.save()
        create_shipments_async()
        self.assertEqual(self.p.shipments.count(), 2)

Ta-da! Of course there are plenty of other tests you could write, but just these four basic tests exercise the code reasonably well. And it would have been virtually impossible to get this level of testing fidelity with the original function. Plus, if one of these tests fails due to a later commit, we can immediately hone in on the cause. The refactoring and tests took less than an hour, and it will save me at least that much worry.

I hope you found this a useful, real world example of refactoring for testability. Love to hear any additional thoughts in the comments!

P.S. This code made it into production yesterday and ran successfully for the first time last night. Phew! Also -- you should sign up for ePantry and never run out of soap or toilet paper ever again.

The No-Frills Guide to PGP on OS X

2013-11-01T00:09:00-07:00

Want to get started using PGP on your Mac, but confused by the morass of professor-doctor style sites with seemingly out-of-date software and plugins? Have no fear! I'll have you up and running in minutes with this handy guide. You'll learn how to use public key servers and how to encrypt and send and receive and decrypt emails, and how to sign and verify messages using GPG. By the time you're done, you might actually find the GPG manual pages for simple tasks illuminating instead of totally incoherent.

Really explaining how public key encryption works or why you'd want to use it is not the goal of this post by a long shot - this is just a simple block-and-tackle tutorial on how to use the stuff. If you want an easy-to-read, non-technical intro to asymmetric cryptography, go read the first section of chapter 10 of Little Brother by Cory Doctorow. But with that said, I'd be remiss if I didn't at least provide...

Part 0 - A Two-Sentence Introduction to Public Key Encryption

Everyone who wants to communicate securely generates their own pair of keys, one of which they publicize, and one of which they keep private. To send someone a secure message, just encrypt your message with the recipient's public key, and that person (and only that person) will be able to decrypt it using their private key.

Got it?

Part 1 - A Quick Disambiguation of PGP, OpenPGP, and GPG

PGP stands for Pretty Good Privacy and is a piece of software that implements the OpenPGP public key (or asymmetric) encryption standard. The PGP implementation is owned by Symantec. GPG is another, free, implementation of OpenPGP that stands for Gnu Privacy Guard. GPG is very common on *nix systems and it's what we'll use here. So basically GPG and PGP are functionally equivalent.

Part 2 - Installing Stuff

We're going to use a Thunderbird (which is a great email client by Mozilla) extension called Enigmail to send and receive GPG emails. Let's get installing!

Get Thunderbird and hook it up to an email account you use. Thunderbird is pretty great, and works well with Gmail (you only need to do the super short instructions in the "configuring your gmail" section - it's that easy). Ok done? Good.
Now we need to install GPG. For OS X, you'll want GPG Suite. It's super easy to install, and will walk you through creating your first GPG key pair just after installation. It will default to an RSA key of 2048 bits, but I'd recommend using 4096. It's quite a bit more secure and really doesn't have any downsides - it's a bit more computationally expensive, but that doesn't matter unless you're using the key for something like SSL. We're just encrypting emails here.
Finally, install Enigmail. This is a Thunderbird extension that provides simple integration with GPG so you don't have to muck about with a bunch of command line tools just to deal with sending and receiving email. There are a few configuration options - odds are, unless you are a character in Cryptonomicon, you don't want to encrypt or sign your messages by default. The official homepage of Engimail makes it look a little long in the tooth but it works great with the latest version of Thunderbird.

Part 3 - Using Key Servers

Now that you have a key pair created, you need to share it with the world - so that anyone who wants to communicate with you securely can encrypt messages to you with your public key. There are a number of organizations that maintain public keyservers to do just this. Key servers are just searchable directories of public keys. I use the MIT's because Phil Zimmerman, the inventor of PGP, is an MIT guy, and the server has been around a heck of a long time.

Go ahead and search for me. That's my key!

If you click on the key, you'll actually see the long block of gibberish that is my public RSA key. But you don't even have to interact with that - the GPG Suite makes it super easy to publish and install keys public keys. In the GPG Keychain Access tool (under Applications on your Mac), go to Apple > Preferences and point to the MIT key server:

Now you can publish your public key to the server by right-clicking your key and hitting "Send public key to Keyserver":

Now go search for your email on MIT's servers and you can find your public key! Not only that, MIT's key server will propagate your key to other keyservers all around the world.

Installing keys is just as easy. You can install mine by going to Key > Retrieve From Keyserver and putting in my key ID from MIT's server (0x6f0eff6b2e0593ad). That's me! And now the world can find your public key just as easily.

Part 4 - Sending & Receiving Encrypted Messages

With Engimail installed, this is really pretty easy. Open Thunderbird and compose a new message. You'll see the OpenPGP drop down menu at the top, and you can elect to encrypt the message.

When you go to send the message, you'll of course have to select the public key with which to encrypt the message. Enigmail will detect the key automatically if you have a key with a matching email address on file with your GPG Keychain already, but if not you'll have to select one (or, more likely, go retrieve and install the correct one from a keyserver).

So if you write me a message like this:

And choose to encrypt it with my private key, here is the email that actually gets sent (note that the subject line is NOT encrypted):

And if I open it in Thunderbird, Engimail detects the encryption, prompts for my password, decrypts it, and lets me know that it's decrypted a message:

Couldn't be easier! The harder thing is actually finding someone who also knows how to send and receive encrypted emails. Hah hah.

Part 5 - Signing & Verifying Emails

Besides keeping communication private, public key cryptography has another superpower - identify verification. The properties of the private/public key pairs means that, not only can someone else use your public key to encrypt a message that is only decryptable by you, but you can encrypt a message using your private key that can only be decrypted by your public key. Now, you may be asking yourself "why is that useful? Why bother encrypting a message that anyone can then read?". Well, if the message can be decrypted using your public key, then it must have been signed using your private key - which means you must have sent the message! Thus we can exploit public key cryptography for identity verification, as well as secret-message sending.

A signed message looks like this:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Really! -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) Comment: GPGTools - https://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCgAGBQJSmOyIAAoJEG8O/2suBZOtmSUQAMSEbn13rjey0L9trG/Qc3eI DEvbnl1Vkx+37d7o8GVtwqmD5uwfh2RphyRq/l1ML3fz00pFzmMH7mfNib6BVZ9U nMXkN+r3x2VUUCNcYbn6i3lXWkxNwYiUIiUQpgwj9DKKv3n7ujNxC/u/1d+dcVK4 hlrf1oHCMMPIcqyQKDRKkPNOB9LGm1Y1KlKFKA6C/ElEAn/48kJKlCKZA55VJSk2 TuxutrduaPOkPjuY9zBGHWlcfF5d1CtZaQALPxjcBbS0z9mSW4vqJRO4kRUY7SAV jWFBfmPKNVPcZQTqvPd6CfqyQqmgCnl1fIHThHviWAMbX3GdGjsscRSzC3DiJRgZ cMcXLqjlqWYaml/6Iq8V8+Azk6Ph2ORxZOsKDiOAz+VwZRQwlyjMu9SSkA/VABEY dN7OrNZZwTwz2A1/QH/SHNvRVAB3kLRpzJTaLJsuFBeh5aycjjbhETHtYccYsxDf rmuGnnfxBrglkbBaYExxzKutaE/yVeFLSegO9clxa2biSk31X51kOjS/2Vy8UQgd iVRNQb/3ArfjOFiQvXIylGkJS0aiVmJXrkEOyiSzy5h0JGxpa2T4JWZ9VyrpGzLx 8PJUPPcYUGcfTEcB0dRvBC7/GpTn/ChEcMBfrPAqI+srsPG0CIBp+aIDAQEsSH61 tWVnIgeDuHCOFrNrB3bK =SYaF -----END PGP SIGNATURE-----

Under the hood, the signature was generated by hashing the message contents (in this case you can see Enigmail inserted the hashing algorithm it used - SHA512), then using the sender's private key to encrypt the hash along with a timestamp. The recipient can then verify the message by decrypting the signature blog, re-hashing the contents, and comparing. Not only is the sender's identity verified, but so are the message contents and the sending time. Cool!

Sending and receiving signed messages through Engimail is just as easy as sending and receiving encrypted messages - just select the option when sending, and Enigmail handles everything for you. Signed messages are automatically verified. Here's what the above message looks like when viewed in Engimail:

Sometimes you might want to verify blocks of text (or an email you received through a client other than Thunderbird). That's really simple with GPG as well, though you'll have to go to the command line. To try it out for yourself, save the above signed message as test.txt and then run the following from a terminal window:

Ta-da! Piece of cake.

A Quick Digression About Trusted Signatures

You'll see an ominous WARNING in the gpg output above, letting you know that this is not a trusted signature. This simply means that you have not indicated that you trust the source of the public key you used to decrypt the message. For instance, what if I am not Chris Clark at all? But some nefarious impersonator who has broken into Chris' blog and provided a public key in this post that is not Chris' at all? What then?? Who can we trust?

Turns out this is actually a pretty hard problem to solve. The PGP community has a concept called the web of trust, which you can read about on your own. Another approach is key signing parties. Ultimately you will need to make the trust determination on your own. You can then set your trust level for each public key in the GPG Keychain tool. By default all keys except your own are "undefined", and this warning won't go away until you've indicated you have "ultimate" trust in the key.

Back to verification

For kicks, try modifying the message slightly - replace my "Really!" with "Really." and re-run the verification (also note that I've now set my own key back to "Ultimate" trust so the warning is gone):

And that's it! You know how to use the GPG tool to communicate securely. There are all sorts of intricacies and details that are loads of fun to learn about, and you should now have a bit of a foundation to go exploring. Maybe those manual pages aren't so cryptic now after all. Good luck, and leave a comment if you'd like to exchange some encrypted emails with yours truly. You know where to find me.

Form Over Function & The New iPhone

2013-09-12T05:57:00-07:00

I probably found this more profound than it really is, but Matt Buchanan said it well in the New Yorker, regarding the latest iPhone announcements:

Fundamental technology, like manufacturing processes for processors and imaging sensors and displays, have evolved to the point that the basic shape and sense of a phone—a thin rectangle with a four-to-five-inch high-resolution touch screen stuffed with a variety of sensors—is determined now largely based on its merits rather than its outright technical limitations, much the same way that the basic shape of a knife is defined by its function rather than our ability to produce it.

It's amazing to think that this is true; that the technological achievements of these devices are perhaps being held back more by the size and shape of our hands than by our manufacturing capabilities.

Declarative Approach to Nesting Backbone Models

2013-08-01T23:57:00-07:00

Backbone doesn't have great (any?) support for nested models. Here's my approach. I think it's kind of fun (we get to write recursive functions!), and hopefully useful. I'll report back when I've lived with it for a bit longer.

Here's the problem: We have a Backbone model with attributes that ought to be other Backbone models, and some of those models' attributes ought to also be Backbone models. We want Backbone models all the way down! But when our REST endpoint returns data from the server, Backbone doesn't know that and hydrates only the top-level model to a first-class Backbone object. So we end up with ugly code inside _.each loops calling new App.Models.MyModel(data) all over the place. It's gross.

More concretely, after we fetch() a model (in this example, we'll use a Customer model with some nested data), we typically get a data structure back that looks like this:

customer [App.Models.Customer]
  attributes : [Object]
    name : (String)
    email : (String)
    address : [Object]
    shipments : [Array]
      0 : [Object]
        date : (String)
        total : (Float)
        items : [Array]
          0 : [Object]
          1 : [Object]
      1 : [Object]
        date : (String)
        total : (Float)
        items : [Array]
          0 : [Object]
          1 : [Object]

But I actually have a bunch of Models and Collections that correspond to various pieces of my Customer model, so what I really want is this:

customer : [App.Models.Customer]
  attributes : [Object]
    name : (String)
    email : (String)
    address : [App.Models.Address]
    shipments : [App.Collections.Shipments]
      models : [Array]
        0 : [App.Models.Shipment]
          attributes : [Object]
            date : (String)
            total : (Float)
            items : [App.Collections.ShipmentItems]
              Models : [Array]
                0 : [App.Models.ShipmentItem]
                1 : [App.Models.ShipmentItem]
        1 : [App.Models.Shipment]
          attributes : [Object]
            date : (String)
            total : (Float)
            items : [App.Collections.ShipmentItems]
              models : [Array]
                0 : [App.Models.ShipmentItem]
                1 : [App.Models.ShipmentItem]

In other words, I want to walk down my root object, and map models and collections onto it where appropriate (we aren't trying to turn EVERYTHING into a Backbone object, just the objects we have models and collections for). You can of course just write a bunch of loops and do it by hand, but I've tried to great a more general purpose method that can be reused across an app. Basically, each model can provide a "map" of how it should be rehydrated into a complete, nested model when it returns from the server. So let's write that (all examples are in Coffeescript):

class App.Models.Customer extends Backbone.Model

  map: () ->
    [
      { name: "address", obj: App.Models.Address },
      {
        name: "shipments", obj: App.Collections.Shipments,
        children: [
          { name: "items", obj: App.Collections.ShipmentItems }
        ]
      },
    ]

Pretty simple! Using standard JSON we declare which attributes we want to turn into Backbone objects, and provide an reference to the relevant class. For deeper nesting, we just specify the child mappings in the same manner. With this simple pattern, we can easily write arbitrarily complex mappings, with deep nesting. Also, Backbone is helpful during the Collection instantiation process and converts the contents of an array into Backbone models, which is why we can stop the map at the Collections level (although we could go deeper).

Note also that map is an anonymous function, rather than just a property. This is because we want to evaluate map at the time we hydrate the object to ensure that all of the relevant Backbone types are loaded.

Now we need a way to walk the Customer model and apply the mapping. We'll write a helper function called "hydrate" for that, which will recursively walk the map and hydrate the objects:

App.hydrate = (root, map) =>
  if root instanceof Backbone.Model
    App.hydrate(root.attributes, map)
  else
    _.each map, (field) ->
      if field.children
        root[field.name] = App.hydrate(root[field.name], field.children)
      if _.isArray root
        _.each root, (i) -> i[field.name] = new field.obj(i[field.name])
      else
        root[field.name] = new field.obj(root[field.name])
  root

Let's look at this in a bit more depth because I am barely smart enough to write recursive functions and it will give me confidence that I did it correctly if I can explain it to you.

First, if the root object is already a Backbone object, we'll walk the attributes instead of the Model itself. This is mostly a convenience thing so that we can call hydrate(model) instead of hydrate(model.attributes) but it also adds a bit of robustness in case you're doing something weird and call hydrate on a model where some, but not all, of the nest models are already Backbone objects. This'll still work and just hydrate the ones that still need it. I don't have a guard here against already-hydrated Collections, but it would be easy enough to add one.

The function then walks through each field in the map, recursively calling hydrate if there are any child maps present. Once we've reach the bottom of the map, the function checks whether the object is an array (in which case each item needs to be hydrated), or just a simple object (in which case just the target field needs to be hydrated).

This version of the function isn't tail-recursive because JavaScript VMs don't have tail-call optimization so there's not point. It's easy to flip the order of the field.children check though, and make it tail recursive for when ECMAScript 6 comes out. The tail-recursive version of the function is just a little tricker to explain, so I opted for this one.

Ok - got all that? The next step is to actually call the darn thing! Because I'm in an auto-magical mood, we'll stick on the model's parse() method. Parse fires after every fetch() and save() of the model, so we're guaranteed to get the hydrated version back after every server call. The full customer class thus looks like this:

class App.Models.Customer extends Backbone.Model

  map: () ->
      ...

  parse: (response) ->
      App.hydrate response, @map()

And voila! Every time we sync from the server, the model will hydrate itself! Neato!

Edit: After writing this, I found this answer on Stackoverflow, which is very similar, and I like a lot. In some situations I like my approach because there is a single map of the entire nesting model in one place, on the root model, so it's very quick to get your head around the model relationships. On the other hand, rycfung's approach is nicely encapsulated and the parsing code is much simpler. And there is a lot to be said for simple code when you are up late at night debugging recursive functions.

How to Write a Bug Report

2013-07-11T00:32:00-07:00

Writing good bug reports is the difference between actually seeing your bug get fixed and sending protracted emails over the course of a week convincing a developer that there is, in fact, a bug. By far the most important information you can provide in a bug report are clear reproduction steps. Here's my guidance on how to write these:

Pretend you are writing to someone who has a very basic understanding of the product.
Now also pretend that this person is an idiot.
Tell them, step-by-step, how to reproduce the issue.

So good reproduction steps look like this:

Log in as user test@test.com with password "foo" on the live website

View the dashboard

Note the dollar amount in the "Your subscription plan" area in the lower left

Hit the "pay" button next to that amount

Expected Result "Today's Payment" is populated in step 1 of the checkout form

Actual Result "Today's Payment" is blank

Yes this can be a little tedious, but if you get in the habit of doing it, it only takes like 30 seconds more per bug, and if it takes longer it's because it's forcing you to actually reproduce the thing consistently, which is the whole point in the first place.

And using this numbered, step-wise, idiot-proof format will cut down on the back-and-forth between you and whoever is fixing the bug by an order of magnitude. So it's ultimately a huge time saver and a great habit to get into. You life will contain 25% fewer bugs and developers will hug you.

Bonus tip: Whenever possible, provide screenshots!

Changing Our Development Process

2013-07-11T00:22:00-07:00

This is a post about change management at a start-up.

Kaggle, at just under 20 people, is an interesting size because we are right on the cusp of being able to wing it in terms of process and communication, and needing more formal processes to make sure changes and plans are communicated to everyone who needs to know.

For a while our engineering team was just winging it - we used Asana (a sophisticated to-do list application) to assign and track tasks, but didn't have much estimation rigor, work would get scheduled "just in time", and priorities were constantly changing. While this roughly worked, and product was going out the door, we could have been doing better on a few fronts:

Productivity. With better prioritization and specification, task-switching time drops, and there are fewer questions to answer while an item is in flight.
Morale. With an ad-hoc process it's very easy to agree to deliver a new feature without a clear understanding of what's getting traded off. This leads to misset expectations, or poorly communicated changes and stresses everyone out.
Predictability. People are crummy at estimating how long complex tasks will take. So it's no surprise that, lacking good estimation techniques, we were consistently inaccurate estimating delivery dates for features.
Transparency. Asana is great for many things, but it was ultimately too flexible and inconsistent for anyone to look to it as a single source of truth for all development activities. So instead, they would ask me or individual engineers when something is going to ship.

We wanted a process that addressed these issues, without creating a bunch of bureaucracy that is anathema to start-ups (or, as one of our engineers put it, "better to manage, but worse"). I'll detail the process itself in a future post, but wanted to discuss here how we managed the change.

There were two basic approaches we could take:

Make incremental changes, learning along the way, and evolve a process the whole team was happy with. We could research and try a few different tools, experiment with stand-ups, and ultimately figure out what makes the most sense for our team
Make a sudden, larger, opinionated change and tweak from there. Do a big-bang roll-out of an agile development process and incorporate the agile tools I've come to really like over the years.

I went for option 2. Management strong-arming is not likely to go over well with any engineering team, but I thought it actually made sense in this case for a few reasons:

The developers on the team had either good experiences with agile, or no experience with agile; there was no scar tissue.
While our company culture is contemplative, flat, and autonomous, we also place a lot of value on just trying stuff quickly.
I had earned enough political capital as an engineering manager that I the team would give it a fair shake even if I was being a dictatorial jerk. In fact, I had tried to do something like this much earlier in my tenure at Kaggle and the team pushed back because I hadn't earned credibility yet.

And crucially, during the roll-out meeting, I explained that yes, I am being unilateral here, but here's why I think it makes sense anyway, and here is the meeting we are having in 3 weeks to evaluate how you like it, and at that same meeting you can decide to blow it all up if you hate it.

The final piece of the puzzle was to make it feel like a real change. When changing largely invisible things like process there's the risk that, in spite of the change, people roughly go back to doing roughly what they were doing before, just with different window dressing. To make it feel tangible and different, we mounted a big TV on the wall that displayed our current sprint, rearranged the floor plan to give a clear sense of space to the team, and made sure out our tracking tool (Pivotal) was integrated with our company chat room on day one so everyone would seem the stream of activity generated by the new process.

And bang! On Friday we were in ad-hoc winging-it land, and on Monday we were starting our first sprint, sitting in our first sizing meeting, with Fibonacci planning poker cards in front of us.

Getting Started with Pandas - Predicting SAT Scores for New York City Schools

2013-01-23T01:20:00-08:00

An Introduction to Pandas

This tutorial will get you started with Pandas - a data analysis library for Python that is great for data preparation, joining, and ultimately generating well-formed, tabular data that's easy to use in a variety of visualization tools or (as we will see here) machine learning applications. This tutorial assumes a solid understanding of core Python functionality, but nothing about machine learning or Pandas.

Goals

Using data from [NYC Open Data] (https://data.cityofnewyork.us/), build a unified, tabular dataset ready for use with machine learning algorithms to predict student SAT scores on a per school basis.
Learn and use the Pandas data analysis package.
Learn how data is typically prepared for machine learning algorithms (ingestion, cleaning, joining, feature generation).

First, let's ingest the data and get the lay of the land. You can download the data sets referenced below from NYC Open Data, or directly download a zip file with the relevant data.

import pandas as pd
import numpy as np

# Load the data
dsProgReports = pd.read_csv('C:/data/NYC_Schools/School_Progress_Reports_-_All_Schools_-_2009-10.csv')
dsDistrict = pd.read_csv('C:/data/NYC_Schools/School_District_Breakdowns.csv')
dsClassSize = pd.read_csv('C:/data/NYC_Schools/2009-10_Class_Size_-_School-level_Detail.csv')
dsAttendEnroll = pd.read_csv('C:/data/NYC_Schools/School_Attendance_and_Enrollment_Statistics_by_District__2010-11_.csv')[:-2] #last two rows are bad
dsSATs = pd.read_csv('C:/data/NYC_Schools/SAT__College_Board__2010_School_Level_Results.csv') # Dependent

dsSATs

Output:

<class 'pandas.core.frame.DataFrame'>
Int64Index: 460 entries, 0 to 459
Data columns:
DBN                      460  non-null values
School Name              460  non-null values
Number of Test Takers    460  non-null values
Critical Reading Mean    460  non-null values
Mathematics Mean         460  non-null values
Writing Mean             460  non-null values
dtypes: object(6)

Outline

Pandas has read the data files without issue. Next let's create a rough map of where we are going.

We have five datasets here, each with information about either schools or districts. We're going to need to join all of this information together into a tabular file, with one row for each school, joined with as much information we can gather about that school & its district, including our dependent variables, which will be the mean SAT scores for each school in 2010.

Drilling down one level of detail, let's look at the dataset dsSATs, which contains the target variables:

dsSATs

Output:

<class 'pandas.core.frame.DataFrame'>
Int64Index: 460 entries, 0 to 459
Data columns:
DBN                      460  non-null values
School Name              460  non-null values
Number of Test Takers    460  non-null values
Critical Reading Mean    460  non-null values
Mathematics Mean         460  non-null values
Writing Mean             460  non-null values
dtypes: object(6)

Target Variable and Joining Strategy

We are going to build a dataset to predict Critical Reading Mean, Mathematics Mean, and Writing Mean for each school (identified by DBN).

After digging around in Excel (or just taking my word for it) we identify the following join strategy (using SQL-esque pseudocode):

dsSATS join dsClassSize on dsSATs['DBN'] = dsClassSize['SCHOOL CODE']
join dsProgReports on dsSATs['DBN'] = dsProgReports['DBN']
join dsDistrct on dsProgReports['DISTRICT'] = dsDistrict['JURISDICTION NAME']
join dsAttendEnroll on dsProgReports['DISTRICT'] = dsAttendEnroll['District']

Now that we have the strategy identified at a high level, there are a number of details we have to identify and take care of first.

Primary Keys - Schools

Before we can join these three datasets together, we need to normalize their primary keys. Below we see the mismatch between the way the DBN (school id) field is represented in the different datasets. We then write code to normalize the keys and correct this problem.

pd.DataFrame(data=[dsProgReports['DBN'].take(range(5)), dsSATs['DBN'].take(range(5)), dsClassSize['SCHOOL CODE'].take(range(5))])

Output:

	0	1	2	3	4
DBN	01M015	01M019	01M020	01M034	01M063
DBN	01M292	01M448	01M450	01M458	01M509
SCHOOL CODE	M015	M015	M015	M015	M015

#Strip the first two characters off the DBNs so we can join to School Code
dsProgReports.DBN = dsProgReports.DBN.map(lambda x: x[2:])
dsSATs.DBN = dsSATs.DBN.map(lambda x: x[2:])

#We can now see the keys match
pd.DataFrame(data=[dsProgReports['DBN'].take(range(5)), dsSATs['DBN'].take(range(5)), dsClassSize['SCHOOL CODE'].take(range(5))])

	0	1	2	3	4
DBN	M015	M019	M020	M034	M063
DBN	M292	M448	M450	M458	M509
SCHOOL CODE	M015	M015	M015	M015	M015

Primary Keys - Districts

We have a similar story with the district foreign keys. Again, we need to normalize the keys. The only additional complexity here is that dsProgReports['DISTRICT'] is typed numerically, whereas the other two district keys are typed as string. We do some type conversions following the key munging.

#Show the key mismatchs
#For variety's sake, using slicing ([:3]) syntax instead of .take()
pd.DataFrame(data=[dsProgReports['DISTRICT'][:3], dsDistrict['JURISDICTION NAME'][:3], dsAttendEnroll['District'][:3]])

Output:

	0	1	2
DISTRICT	1	1	1
JURISDICTION NAME	CSD 01 Manhattan	CSD 02 Manhattan	CSD 03 Manhattan
District	DISTRICT 01	DISTRICT 02	DISTRICT 03

#Extract well-formed district key values
#Note the astype(int) at the end of these lines to coerce the column to a numeric type
import re
dsDistrict['JURISDICTION NAME'] = dsDistrict['JURISDICTION NAME'].map(lambda x: re.match( r'([A-Za-z]*\s)([0-9]*)', x).group(2)).astype(int)
dsAttendEnroll.District = dsAttendEnroll.District.map(lambda x: x[-2:]).astype(int)

#We can now see the keys match
pd.DataFrame(data=[dsProgReports['DISTRICT'][:3], dsDistrict['JURISDICTION NAME'][:3], dsAttendEnroll['District'][:3]])

Output:

	0	1	2
DISTRICT	1	1	1
JURISDICTION NAME	1	2	3
District	1	2	3

Additional Cleanup

At this point we could do the joins, but there is messiness in the data still. First let's reindex the DataFrames so the semantics come out a bit cleaner. Pandas indexing is beyond the scope of this tutorial but suffice it to say it makes these operations easier.

#Reindexing
dsProgReports = dsProgReports.set_index('DBN')
dsDistrict = dsDistrict.set_index('JURISDICTION NAME')
dsClassSize = dsClassSize.set_index('SCHOOL CODE')
dsAttendEnroll = dsAttendEnroll.set_index('District')
dsSATs = dsSATs.set_index('DBN')

Let's take a look at one of our target variables. Right away we see the "s" value, which shouldn't be there.

We'll filter out the rows without data.

#We can see the bad value
dsSATs['Critical Reading Mean'].take(range(5))

Output:

DBN
M292    391
M448    394
M450    418
M458    385
M509      s
Name: Critical Reading Mean

Now we filter it out

#We create a boolean vector mask. Open question as to whether this semantically ideal...
mask = dsSATs['Number of Test Takers'].map(lambda x: x != 's')
dsSATs = dsSATs[mask]
#Cast fields to integers. Ideally we should not need to be this explicit.
dsSATs['Number of Test Takers'] = dsSATs['Number of Test Takers'].astype(int)
dsSATs['Critical Reading Mean'] = dsSATs['Critical Reading Mean'].astype(int)
dsSATs['Mathematics Mean'] = dsSATs['Mathematics Mean'].astype(int)
dsSATs['Writing Mean'] = dsSATs['Writing Mean'].astype(int)

#We can see those values are gone
dsSATs['Critical Reading Mean'].take(range(5))

Output:

DBN
M292    391
M448    394
M450    418
M458    385
M515    314
Name: Critical Reading Mean

Feature Construction

dsClassSize will be a many-to-one join with dsSATs because dsClassSize contains multiple entries per school. We need to summarize and build features from this data in order to get one row per school that will join neatly to dsSATs.

Additionally, the data has an irregular format, consisting of a number of rows per school describing different class sizes, then a final row for that school which contains no data except for a number in the final column, SCHOOLWIDE PUPIL-TEACHER RATIO.

We need to extract the SCHOOLWIDE PUPIL-TEACHER RATIO rows, at which point we'll have a regular format and can build features via aggregate functions. We'll also drop any features that can't be easily summarized or aggregated and likely have no bearing on the SAT scores (like School Name).

#The shape of the data
print dsClassSize.columns
print dsClassSize.take([0,1,10]).values

Output:

array([BORO, CSD, SCHOOL NAME, GRADE , PROGRAM TYPE,
       CORE SUBJECT (MS CORE and 9-12 ONLY),
       CORE COURSE (MS CORE and 9-12 ONLY), SERVICE CATEGORY(K-9* ONLY),
       NUMBER OF CLASSES, TOTAL REGISTER, AVERAGE CLASS SIZE,
       SIZE OF SMALLEST CLASS, SIZE OF LARGEST CLASS, DATA SOURCE,
       SCHOOLWIDE PUPIL-TEACHER RATIO], dtype=object)
[[M 1 P.S. 015 ROBERTO CLEMENTE 0K GEN ED - - - 1.0 21.0 21.0 21.0 21.0 ATS
  nan]
 [M 1 P.S. 015 ROBERTO CLEMENTE 0K CTT - - - 1.0 21.0 21.0 21.0 21.0 ATS
  nan]
 [M 1 P.S. 015 ROBERTO CLEMENTE nan nan nan nan nan nan nan nan nan nan nan
  8.9]]

Extracting the Pupil-Teacher Ratio

#Take the column
dsPupilTeacher = dsClassSize.filter(['SCHOOLWIDE PUPIL-TEACHER RATIO'])
#And filter out blank rows
mask = dsPupilTeacher['SCHOOLWIDE PUPIL-TEACHER RATIO'].map(lambda x: x > 0)
dsPupilTeacher = dsPupilTeacher[mask]
#Then drop from the original dataset
dsClassSize = dsClassSize.drop('SCHOOLWIDE PUPIL-TEACHER RATIO', axis=1)

#Drop non-numeric fields
dsClassSize = dsClassSize.drop(['BORO','CSD','SCHOOL NAME','GRADE ','PROGRAM TYPE',\
'CORE SUBJECT (MS CORE and 9-12 ONLY)','CORE COURSE (MS CORE and 9-12 ONLY)',\
'SERVICE CATEGORY(K-9* ONLY)','DATA SOURCE'], axis=1)

#Build features from dsClassSize
#In this case, we'll take the max, min, and mean
#Semantically equivalent to select min(*), max(*), mean(*) from dsClassSize group by SCHOOL NAME
#Note that SCHOOL NAME is not referenced explicitly below because it is the index of the dataframe
grouped = dsClassSize.groupby(level=0)
dsClassSize = grouped.aggregate(np.max).\
    join(grouped.aggregate(np.min), lsuffix=".max").\
    join(grouped.aggregate(np.mean), lsuffix=".min", rsuffix=".mean").\
    join(dsPupilTeacher)

print dsClassSize.columns

Output:

array([NUMBER OF CLASSES.max, TOTAL REGISTER.max, AVERAGE CLASS SIZE.max,
       SIZE OF SMALLEST CLASS.max, SIZE OF LARGEST CLASS.max,
       NUMBER OF CLASSES.min, TOTAL REGISTER.min, AVERAGE CLASS SIZE.min,
       SIZE OF SMALLEST CLASS.min, SIZE OF LARGEST CLASS.min,
       NUMBER OF CLASSES.mean, TOTAL REGISTER.mean,
       AVERAGE CLASS SIZE.mean, SIZE OF SMALLEST CLASS.mean,
       SIZE OF LARGEST CLASS.mean, SCHOOLWIDE PUPIL-TEACHER RATIO], dtype=object)

Joining

One final thing before we join - dsProgReports contains distinct rows for separate grade level blocks within one school. For instance one school (one DBN) might have two rows: one for middle school and one for high school. We'll just drop everything that isn't high school.

And finally we can join our data. Note these are inner joins, so district data get joined to each school in that district.

mask = dsProgReports['SCHOOL LEVEL*'].map(lambda x: x == 'High School')
dsProgReports = dsProgReports[mask]

final = dsSATs.join(dsClassSize).\
join(dsProgReports).\
merge(dsDistrict, left_on='DISTRICT', right_index=True).\
merge(dsAttendEnroll, left_on='DISTRICT', right_index=True)

Additional Cleanup 2

We should be in a position to build a predictive model for our target variables right away but unfortunately there is still messy data floating around in the dataframe that machine learning algorithms will choke on. A pure feature matrix should have only numeric features, but we can see that isn't the case. However for many of these columns, the right approach is obvious once we've dug in.

final.dtypes[final.dtypes.map(lambda x: x=='object')]

Output:

School Name                      object
SCHOOL                           object
PRINCIPAL                        object
PROGRESS REPORT TYPE             object
SCHOOL LEVEL*                    object
2009-2010 OVERALL GRADE          object
2009-2010 ENVIRONMENT GRADE      object
2009-2010 PERFORMANCE GRADE      object
2009-2010 PROGRESS GRADE         object
2008-09 PROGRESS REPORT GRADE    object
YTD % Attendance (Avg)           object

#Just drop string columns.
#In theory we could build features out of some of these, but it is impractical here
final = final.drop(['School Name','SCHOOL','PRINCIPAL','SCHOOL LEVEL*','PROGRESS REPORT TYPE'],axis=1)

#Remove % signs and convert to float
final['YTD % Attendance (Avg)'] = final['YTD % Attendance (Avg)'].map(lambda x: x.replace("%","")).astype(float)

#The last few columns we still have to deal with
final.dtypes[final.dtypes.map(lambda x: x=='object')]

Output:

2009-2010 OVERALL GRADE          object
2009-2010 ENVIRONMENT GRADE      object
2009-2010 PERFORMANCE GRADE      object
2009-2010 PROGRESS GRADE         object
2008-09 PROGRESS REPORT GRADE    object

Ordered Categorical Variables

We can see above that the remaining non-numeric field are grades . Intuitively, they might be important so we don't want to drop them, but in order to get a pure feature matrix we need numeric values. The approach we'll use here is to explode these into multiple boolean columns. Some machine learning libraries effectively do this for you under the covers, but when the cardinality of the categorical variable is relatively low, it's nice to be explicit about it.

gradeCols = ['2009-2010 OVERALL GRADE','2009-2010 ENVIRONMENT GRADE','2009-2010 PERFORMANCE GRADE','2009-2010 PROGRESS GRADE','2008-09 PROGRESS REPORT GRADE']

grades = np.unique(final[gradeCols].values) #[nan, A, B, C, D, F]

for c in gradeCols:
    for g in grades:
        final = final.join(pd.Series(data=final[c].map(lambda x: 1 if x is g else 0), name=c + "_is_" + str(g)))

final = final.drop(gradeCols, axis=1)

#Uncomment to generate csv files
#final.drop(['Critical Reading Mean','Mathematics Mean','Writing Mean'],axis=1).to_csv('C:/data/NYC_Schools/train.csv')
#final.filter(['Critical Reading Mean','Mathematics Mean','Writing Mean']).to_csv('C:/data/NYC_Schools/target.csv')

Modeling

We now have a feature matrix and it's trivial to use it with any number of machine learning algorithms.

from sklearn.ensemble import RandomForestRegressor

target = final.filter(['Critical Reading Mean'])
#We drop all three dependent variables because we don't want them used when trying to make a prediction.
train = final.drop(['Critical Reading Mean','Writing Mean','Mathematics Mean'],axis=1)
model = RandomForestRegressor(n_estimators=100, n_jobs=-1, compute_importances = True)
model.fit(train, target)

predictions = np.array(model.predict(train))
rmse = math.sqrt(np.mean((np.array(target.values) - predictions)**2))
imp = sorted(zip(train.columns, model.feature_importances_), key=lambda tup: tup[1], reverse=True)

print "RMSE: " + str(rmse)
print "10 Most Important Variables:" + str(imp[:10])

Output:

RMSE: 80.13105688
10 Most Important Variables:[('PEER INDEX*', 0.81424747874371173), ('TOTAL REGISTER.min', 0.060086333792196724), ('2009-2010 ENVIRONMENT CATEGORY SCORE', 0.023810405565050669), ('2009-2010 ADDITIONAL CREDIT', 0.021788425210174274), ('2009-2010 OVERALL SCORE', 0.019046860376900468), ('AVERAGE CLASS SIZE.mean', 0.0094882658926829649), ('2009-2010 PROGRESS CATEGORY SCORE', 0.0094678349064146652), ('AVERAGE CLASS SIZE.min', 0.0063723026953534942), ('2009-2010 OVERALL GRADE_is_nan', 0.0057710237481254567), ('Number of Test Takers', 0.0053660239780210584)]

Entering Kaggle Competitions with Google Predict

2013-01-17T16:30:00-08:00

BigML had a great series of posts over the summer pitting some prediction-as-a-service products against each other. One of those was the Google Predict API. I thought it might be fun to enter a Kaggle competition using the API and see how it did against some of the world's top data scientists.

It turns out this was a terrible, terrible waste of time.

If you are expecting a rigorous analysis of the Google Predict API, this post will be a disappointment. In fact, I'll go so far as to ruin the surprise right now: on the Biological Response competition, the Predict API turned in a 0.67245 on the private leaderboard, just edging out the optimized constant value benchmark (in a nutshell - it did badly). It fared a bit better on the Titanic competition, scoring a somewhat-reasonable 0.79426 (tied with 55 other users for 112th place as of this writing).

So instead of focusing on the actual performance of the algorithm, I will instead share some tips and tricks for using the Google Predict API:

Chris' Tips for Using the Google Predict API:

Turn off two factor authentication, unless your idea of a good time is spending a furious evening at home, battling OAuth. It is a formidable enemy.
If you are predicting a binary response, be sure to replace those 0s and 1s in your response column with strings like TRUE and FALSE, or else Google will attempt to perform Magic (stand back!) and decide that your model is actually a regression model and will cheerfully return values like -0.8 and 1.2.
Nice work - now your response column is now filled with TRUE and FALSE strings and you have persuaded Google that you want a categorical response. You also remembered to remove that header row, right? Oh...you didn't? Well then get ready for the excitement of responses like this:
```
"outputMulti": [
   {
     "label": "TRUE",
     "score": 0.2001
   },
   {
     "label": "FALSE",
     "score": 0.7829
   },
   {
     "label": "HEADER_TEXT",
     "score": 0.017
   }
 ],
```
So yeah, remove those headers. And on a related note, have fun writing your own custom parsing code to extract the most likely label from this unsorted list. Because we certainly wouldn't want to pollute the integrity of the response schema with something as handy as providing the predicted label as a first class property.
The thing you are creating is a TrainedModel, not a HostedModel. Just trust me on this - they may have identical predict methods, and be used interchangeably in example code, but you are making TrainedModels and will get errors if you ever accidentally refer to HostedModels.
If you are using Google's Python API wrappers, be sure you are constructing your feature array like this: ["foo","bar",1.23\] and not like this: ["foo","bar","1.23"\]. I made the mistake of reading in my training CSV file as an array of strings, which caused my requests to look like the latter. The result? The Predict API merrily chirps back with a valid, but completely terrible, prediction.
Get a cup of coffee. Google gets angry if you make too many prediction requests in a row and starts throwing errors. So you have to resort to weird throttling tricks if you need more than a few hundred predictions at once. And bulk prediction is not available.

My terrible, horrible code that does this is on github. It's not a particularly usable form and does some...questionable things because I didn't learn some of these lessons until it was too late. But if you are absolutely determined to use the Google Predict API (god help you), it might get you started.

Also, kudos to the BigML engineers who are either way smarter than me and got this working right away, or have a remarkable amount of self-restraint and did not complain one jot.

Extensible, Single-Line Fizzbuzz in a Tweet

2012-11-01T19:55:00-07:00

Yep, it's a fizzbuzz blog post! I can hardly believe I've gone this long without ever doing one.

I was thinking about hiring and what I would do if someone asked me fizzbuzz, and I think I would have used it as an opportunity to show off some engineering practices. Basically, I'd write unit tests and make it extensible. Before I knew it this was no longer a thought exercise and I was fiddling around in IPython Notebook (which is great for this sort of thing) and created very explicit implementation:

swaps = []
swaps.append((3, "fizz"))
swaps.append((5, "buzz"))
for i in xrange(1,101):
    out = ""
    for s in swaps:
        if i % s[0] == 0:
            out += s[1]
    if not len(out):
        out = i
    print out

I like this approach because it's reasonably extensible - just add more tuples to the list of swaps and everything works. Want to print "bazz" for multiples of 4? Just add swaps.append((4, "bazz")).

But it's pretty boring. As I'm always trying to get better at writing idiomatic Python, I started swapping in some more Pythonic constructs:

swaps = []
swaps.append((3, "fizz"))
swaps.append((5, "buzz"))
for i in xrange(1,101):
    out = "".join([s[1] if i % s[0] == 0 else "" for s in swaps])
    print out or i

Now of course, I'm hooked and decide I want to shrink it down to fit in a Tweet. Not terribly difficult, just rename variables, eliminate spaces, and condense variable declarations:

s=[(3,"fizz"),(5,"buzz")]
for i in range(1,101):
    o="".join([x[1] if i%x[0]==0 else "" for x in s])
    print o or i

And I still have 21 characters to spare. But Twitter doesn't support linebreaks so if I were to Tweet it, no one could run it without guessing at the indentation and linebreaks. The semicolon trick almost works but only gets me this far (the line break after the semicolon is still required):

s=[(3,"fizz"),(5,"buzz")];
for i in range(1,101): o="".join([x[1] if i%x[0]==0 else "" for x in s]);print o or i

This is because Python won't let you inline control flow statements. I don't want to move the variable declaration inline because it contradicts any semblance of extensibility still here (I know this is silly, but ya gotta have yer principles), so I have to get creative.

Instead of a loop, a construct like map(lambda, range(1,101)) will work. Again though, a little creativity is in order because "print" is a statement and can't appear inside a lambda (only expressions are allowed). Instead I can use sys.stdout.write. Finally, I added a neat little trick to avoid the if/else (multiplying a string by a boolean value) and wound up with this functioning, extensible, single line Fizzbuzz that fits in a Tweet:

s=[(3,"fizz"),(5,"buzz")];map(lambda i: sys.stdout.write("%s\n" % ("".join((i%x[0]==0)*x[1] for x in s) or i)), range(1,101))

Sure there are shorter versions out there, but I had a lot of fun with my version and picked up a couple additional Python tricks on the way.

(for completeness, if I caved and inlined the variable declaration, I get this, which is pretty darn short)

for i in range(1,101):print "".join((i%x[0]==0)*x[1] for x in [(3,"fizz"),(5,"buzz")]) or i

Rapidly Saving .jpgs in Photoshop

2012-10-17T23:52:00-07:00

Cliffs notes: Now, whenever I hit F2 in Photoshop, I get a high quality jpg of my file saved to the same directory. No more "File->Save As" nonsense every time I want a static version of the image.

What/how/why: I was working on some product mock-ups this morning in Photoshop using layers to show and hide different bits of the mockup, depending on the state of the application. It was time to wire all of the various states together in a clickable mock-up (I use invisionapp.com to do this - highly recommended) so I needed to save off a whole bunch of images from Photoshop, with different layers set to visible. I quickly tired of going to File->Save As->Select Jpg file type->[Type file name]->Hit save. There has to be a better way!

After Googling some options (I am not a PS expert), I decided to create a script that would save the image as a jpg, with a unique name generated from a timestamp. I found some sample code for the various pieces and came up with the script below:

//Cobbled together from:
//http://feedback.photoshop.com/photoshop_family/topics/quick_save_as
//http://www.polycount.com/forum/showthread.php?t=85425

#target photoshop;

if (app.documents.length > 0) {
  var thedoc = app.activeDocument;

  var docName = thedoc.name;
  if (docName.indexOf(".") != -1) {
    var basename = docName.match(/(.*)\.[^\.]+$/)[1];
  } else {
    var basename = docName;
  }

  //getting the location, if unsaved save to desktop;
  try {
    var docPath = thedoc.path;
  } catch (e) {
    var docPath = "~/Desktop";
  }

  var jpegOptions = new JPEGSaveOptions();
  jpegOptions.quality = 9;
  jpegOptions.embedColorProfile = true;
  jpegOptions.matte = MatteType.NONE;

  var filename = docPath + '/' + basename + "-" + getTime() + '.jpg';

  thedoc.saveAs((new File(filename)), jpegOptions, true);
};

function getTime(){
  var currentTime = new Date();

  //Make single-digit mins show up as 6:01 and not 6:1
  var minutes = currentTime.getMinutes();
  if (minutes < 10) {
    minutes = "0" + minutes;
  }

  var timeStamp = currentTime.getFullYear() + "-"
  + (currentTime.getMonth() + 1) + "-"
  + currentTime.getDate() + "-"
  + currentTime.getHours() + "."
  + minutes + "."
  + currentTime.getSeconds() + "."
  + currentTime.getMilliseconds();
  return timeStamp;
}

If you save this as "Quick save to jpg.jsx" to \\Program Files\\Adobe\\Adobe Photoshop CS5.1\\Presets\\Scripts (or /Applications/Adobe Photoshop CS6/Presets/Scripts on Mac OS X), you will then be able to access the script by going to File->Scripts. I went one step further and recorded an action of the script firing and bound it to my f2 key.

Now, whenever I'm in Photoshop, I can just hit f2 and get a .jpg image saved to my current directory, with no dialogs or anything! Neat!

Engineering Practices in Data Science

2012-10-04T17:28:00-07:00

Josh Wills wrote this excellent, pithy definition of a data scientist:

Data Scientists (n.): Person who is better at statistics than any software engineer and better at software engineering than any statistician.

It's certainly true that software engineering and data science are two different disciplines, and for good reason - they require different skills. But as this definition points out, in the same way an artist and an interior painter might share the medium of paint and thus a set of best practices (invest in canvas drop cloths, write down color choices), so do data scientists and software engineers share the medium of code.

Among the Kaggle community there seem to be a few consistent ways that engineering practices intersect with data science. These trends are thrown into particularly sharp relief by the competition format we use at Kaggle, but are present to varying extents in data science teams across industry.

To start, it terrifies me that....

...many data scientists don't use source control.

To those who breathe Git and dream in command line interfaces, it might be hard to believe that many data scientists (smart ones! great ones!) just plain don't use version control. If you ask them they might say something like "Sure I use version control! I email myself every new version of the R script. Plus it's on dropbox."

I can't back it up with hard data, but it seems data scientists who use R, Matlab, SAS, or other statistical programming languages use source control less frequently than people who use Python or Java. And this sort of makes sense - the stats community is often more academically oriented and not as steeped in engineering practices as the Python community, for instance.

And frankly - who can blame data scientists for not using source control? The de facto choice, Git, might be free but is totally obtuse (though R isn't exactly a haven of good design...) and most of the time saving files to disk with clever names seems to work OK. For data scientists, source control on its own just isn't valuable enough to bother with. Contrast this with software engineering: source control isn't just source control but the jumping off point for continuous integration builds, deployments, and code reviews. For data scientists to use source control (and they should - it makes collaboration easier, and mistakes less likely), it has to be more valuable. And I'm optimistic that we will get there as...

....successful data scientists learn the value of good pipeline engineering.

A well-engineered pipeline gets data scientists iterating much faster, which can be a big competitive edge in a Kaggle competition. This is especially true for datasets that require a lot of feature engineering - a crisp pipeline with well defined phases (data ingestion -> feature extraction -> training -> ensembling -> validation) allows disciplined data scientists to try out a lot more ideas than someone with a pile of spaghetti Python code.

I expect we'll soon start to see more open source data science pipelines that couple nicely with popular machine learning libraries and tools. The more tooling and community support we have for good pipelines, the less we'll see...

...good engineering going out the window quickly with elaborate ensembling.

This effect is probably more specific to Kaggle than the others but it's fun and instructive, so I'll write about it anyway.

In the final hours of a competition we sometimes see unholy ensembles of different languages and techniques being mashed together for a final push. It's something we try to discourage by encouraging team formation early on (so a nice pipeline gets built), but it does occasionally happen. In one particularly gruesome case we saw some Java code spawning processes which ran command line shells in order to launch the R executable while the Java program piped data and commands back and forth between the shells. Yikes!

In Closing...

Data scientists are slowly but surely adopting engineering practices that enable them to get the goo of software out of the way so they can focus on valuable data problems. I hope that we'll soon start to see well engineered data pipelines emerge as open source projects that generalize well over particular classes of problems, complement existing data science libraries, and encourage modular development and use of source control.

Or someone writes a killer general-purpose deep learning library and we all just use that.

Getting Started With Python for Data Science

2012-07-20T22:17:00-07:00

As the product manager at Kaggle I'm fortunate enough to hang around with some of the world's top machine learning experts. And working at a place that specializes in running predictive modelling competitions means that I inevitably got the itch to learn some of this stuff myself.

I don't have a super-strong math or machine learning background, but I figured I'm a decent-enough programmer and have all these smart people around me, so I figured I should be able to figure out how to use some off-the-shelf tools to build some predictive models of my own.

I wrote the following tutorial after my experience hacking away on a Kaggle competition for a weekend. I hope it serves to get other programming-literate but machine learning-ignorant people like me up and running, building their own models, and learning data science. So without further ado...

Who is this for and what will I learn?

(Cross posted on Kaggle's blog, No Free Hunch.)

This tutorial assumes some knowledge of Python and programming, but no knowledge whatsoever of data science, machine learning, or predictive modeling (or, heck, even statistics). To the extent there is a target audience, it's probably hacker types who learn best by doing.

All the code from this tutorial is available on github.

You might encounter terms you're not familiar with, but that shouldn't stop you from completing the tutorial. By the end, you won't know a heck of a lot more about data science per se, but you'll have a nice environment set up where you can easily play with lots of different data science tools and even make credible entries to Kaggle competitions. Most importantly you'll be in a great position to experiment and learn more data science.

Here's what you'll learn:

How to install popular scientific and statistical computing libraries for Python
Use those libraries to create a benchmark model and submit it to a competition.
Write your own evaluation function, and learn how to use cross-validation to test out ideas on your own.

Excited? I thought so!

1. Environment Setup

First thing, we'll need a Python environment suitable for scientific and statistical computing. Assuming you already have Python installed (no? Well then get it! Python 2.7 is recommended), we'll need three packages. You should install each in the order they appear here:

numpy - (pronounced num-pie) Powerful numerical arrays. A foundational package for the two packages below.
scipy - (sigh-pie) Scientific, mathematical, and engineering package
scikit-learn - Easy to use machine learning library

Note: 64 bit versions of these libraries can be found here.

Click through the links above for the home pages of each project and get the installation for your operating system or, if you're running Linux, you can install from a package manager (pip). If you're on a Windows machine, it's easiest to install using the setup executables for scipy and scikit-learn rather than installing from a package manager.

I'd also highly recommend to setting up a decent Python development environment. You can certainly execute Python scripts from the command line, but it's a heck of a lot easier to use a proper environment with debugging support. I use PyDev, but even something like IPython is better than nothing.

Now you're ready for machine learning greatness!

2. Your First Submission

The Biological Response competition provides a great data set to get started with because the value to be predicted is a simple binary classifier (0 or 1) and the data is just a bunch of numbers, so feature extraction and selection aren't as important as in some other Kaggle competitions. Download the training and test data sets now. Even though this competition is over, you can still make submissions and see how you compare to the world's best data scientists.

In the code below, we'll use an ensemble classifier called a random forest that often performs very well as-is, without much babysitting and parameter-tweaking. Although a random forest is actually a pretty sophisticated classifier, it's a piece of cake to get up and running with one thanks to sklearn.

Remember: You don't have to understand all of the underlying mathematics to use these techniques. Experimentation is a great way to start getting a feel for how this stuff works. Understanding the models is important, but it's not necessary to get started, have fun, and compete.

Here's the code:

from sklearn.ensemble import RandomForestClassifier
from numpy import genfromtxt, savetxt

def main():
    #create the training & test sets, skipping the header row with [1:]
    dataset = genfromtxt(open('Data/train.csv','r'), delimiter=',', dtype='f8')[1:]
    target = [x[0] for x in dataset]
    train = [x[1:] for x in dataset]
    test = genfromtxt(open('Data/test.csv','r'), delimiter=',', dtype='f8')[1:]

    #create and train the random forest
    #multi-core CPUs can use: rf = RandomForestClassifier(n_estimators=100, n_jobs=2)
    rf = RandomForestClassifier(n_estimators=100)
    rf.fit(train, target)
    predicted_probs = [x[1] for x in rf.predict_proba(test)]

    savetxt('Data/submission.csv', predicted_probs, delimiter=',', fmt='%f')

if __name__=="__main__":
    main()

At this point you should go ahead and actually get this running by plopping the into a new Python script and saving it as makeSubmission.py. You should now have directories and files on your computer like this:

"My Kaggle Folder")
 |
 |---"Data"
 |   |
 |   |---train.csv
 |   |
 |   |---test.csv
 |
 |---makeSubmission.py

Once you've run makeSubmission, you'll also have my_first_submission.csv in your data folder. Now it is time to do something very important:

Submit this file to the bio-response competition

Did you do it? You did it! Great! You could stop here - you know how to make a successful Kaggle entry - but if you're the curious type, I'm sure you'll want to play around with other types of models, different parameters, and other shiny things. Keep reading to learn an important technique that will let you test models locally, without burning through your daily Kaggle submission limit.

3. Evaluation and Cross-Validation

Let's say we wanted to try out sklearn's gradient boosting machine instead of a random forest. Or maybe some simple linear models. It's easy enough to important these things from sklearn and generate submission files, but it's not so easy to compare their performance. It's not practical to upload a new submission every time we make a change to the model - we'll need a way to test things out locally, and we'll need two things in order to do that:

An evaluation function
Cross validation

You'll always need some kind of evaluation function to determine how your models are performing. Ideally, these would be identical to the evaluation metric that Kaggle is using to score your entry. Competition participants often post evaluation code in the forums, and Kaggle has detailed descriptions of the metrics available on the wiki. In the case of the bio-response competition, the evaluation metric is log-loss and user Grunthus has posted a Python version of it. We won't spend too much time on this (read the forum post for more information), but go ahead and save the following into your working directory as logloss.py.

import scipy as sp
def llfun(act, pred):
    epsilon = 1e-15
    pred = sp.maximum(epsilon, pred)
    pred = sp.minimum(1-epsilon, pred)
    ll = sum(act*sp.log(pred) + sp.subtract(1,act)*sp.log(sp.subtract(1,pred)))
    ll = ll * -1.0/len(act)
    return ll

Finally, we'll need data to test our models against. When you submitted your first Kaggle competition entry earlier in this tutorial, Kaggle compared (using log-loss) your answers to the actual real world results (the "ground truth") associated with the test data set. Without access to those answers, how can we actually test our models locally? Cross-validation to the rescue!

Cross-validation is a simple technique that basically grabs a chunk of the training data and holds it in reserve while the model is trained on the remainder of the data set. In case you haven't realized it yet, sklearn it totally awesome and is here to help. It has built in tools to generate cross validation sets. The sklearn documentation has a lot of great information on cross-validation.

The code below creates 10 cross-validation sets (called folds), each with 10% of the training data set held in reserve, and tests our random forest model against that withheld data.

from sklearn.ensemble import RandomForestClassifier
from sklearn import cross_validation
import logloss
import numpy as np

def main():
    #read in  data, parse into training and target sets
    dataset = np.genfromtxt(open('Data/train.csv','r'), delimiter=',', dtype='f8')[1:]
    target = np.array([x[0] for x in dataset])
    train = np.array([x[1:] for x in dataset])

    #In this case we'll use a random forest, but this could be any classifier
    cfr = RandomForestClassifier(n_estimators=100)

    #Simple K-Fold cross validation. 5 folds.
    cv = cross_validation.KFold(len(train), k=5, indices=False)

    #iterate through the training and test cross validation segments and
    #run the classifier on each one, aggregating the results into a list
    results = []
    for traincv, testcv in cv:
        probas = cfr.fit(train[traincv], target[traincv]).predict_proba(train[testcv])
        results.append( logloss.llfun(target[testcv], [x[1] for x in probas]) )

    #print out the mean of the cross-validated results
    print "Results: " + str( np.array(results).mean() )

if __name__=="__main__":
    main()

Note that your cross-validated results might not exactly match the score Kaggle gives you on the model. This could be for a variety of (legitimate) reasons: random forests have a random component and won't yield identical results every time; the actual test set might deviate from the training set (especially when the sample size is fairly low, like in the bio-response competition); the evaluation implementation might differ slightly. Even if you are getting slightly different results, you can compare model performance locally and you should know when you have made an improvement.

That's a wrap!

That's it! We're done! You now have great tools at your disposal, and I expect to see you at the top of the leaderboard in no time!

Tale of the Tape (The Indentation Apocalypse)

2012-07-01T22:13:00-07:00

BRUCE "Emacs" BUFFER:: Now entering the Kill Ring in the blue trunks, with a reach of approximately three-quarters of an inch, hailing from the 9th spot in the ASCII table, it's TAAAABBBBBB!

BRUCE "Emacs" BUFFER: In the opposite corner, wearing white, you know him well, don't-call-him-32-he-prefers-hexidecimal-number-20...it's SPAAACCEEEEEE!

COMMENTATOR 1: Well folks, have we got a treat for you tonight! They're calling it the Indentation Apocalypse. These are two guys - they've fought before - and let me tell you - there is no love lost between these two.

COMMENTATOR 2: You said it. While these guys are ostensibly fighting for the title, you know their real focus is just on beating the other guy. This one's personal, and should be a heck of a fight.

C1: It almost blew up at the pre-fight conference. Tab got right in the face of Space, and accused Space of being semantically inferior.

C2: Oof, brutal stuff. But semantic arguments don't mean much in the ring.

REFEREE: (shouting, to be heard over the crowd): Let's have a good, clean fight, keep everything above the ¶.

C2: And we're off!

C1: Listen to the crowd as Tab struts his stuff around the ring! He's looking great as he sizes up Space - he really worked on his configurability for this one. Such an innovator.

C2: You know, that's a good point. As a fighter, he's a real chameleon - you ask one fan, he'll tell you he's 3/4", but another will tell you he's a full 1.5". I guess that's a reason he's so popular with the crowd.

C1: Yeah, but Space is so consistent and OH! He lands four crisp jabs to Tab's left margin.

C2: Four jabs from Space is about as good as one big right from Tab.

C1: That's true, but Tab is so quick to recover after one of those colossal right hands, he just takes one smooth step back, and he's out of range. Space's footwork is a mess after those jabs - sometimes it seems like he's gotta take a step back for every punch he throws!

C2: I agree. The efficiency edge definitely goes to Tab. At a quarter of the weight as well, there's just no question.

C1: But year after year, space just delivers the same performance.

C2: And he's versatile - remember, he's had a lot of international success, and in some of smaller leagues, which have slightly different rules. His techniques really seem to work equally well everywhere.

C1: In contrast, I'm sure many fans remember Tab's embarrassing Olympic boxing trials. His inconsistencies really hurt him. The coach got so frustrated trying to train him alongside the other team members, he eventually had to kick him out of the program.

C2: Sometimes it's best to stick with the basics.

...rounds pass...

C1: Neither fighter has been able to land the punch they're looking for. The crowd's frustrated - they came here tonight to see a knockout.

C2: Am I glad I'm not a judge tonight. The scores are even on the cards. If it goes to a decision, it's going to come down to this last round. This crowd won't be happy whichever way it goes.

C1: Ringside is already buzzing with talks of a rematch. This fight may be almost over, but it's a rivalry for the ages, and we haven't seen the last of these two.

Four Fun Facts From Big Data

2012-06-29T03:00:00-07:00

The credit card industry's term of art for false positives in fraud detection is "insult rate". If you've ever swiped your card at a shop and had it declined, you understand why.
A large auto insurance company has discovered that if a household owns two identical cars (same make, same model), that household's driving risk is very low. They have termed this "the boring factor".
Cable companies have trouble figuring out whether you are actually watching TV - the box might be on, but the TV is off. But by inspecting remote control button presses and using predictive models, they can figure it out. If you hit the volume button, for instance, you're almost certainly watching. And yes, that means every button press is probably being logged.
Look for paginated articles on news websites to start disappearing. Paginating articles is a way for content providers to "manufacture capacity" (ie. create pageviews against which they can sell advertising). But new display advertising technologies only register an ad impression when the ad is actually viewed (so ads at the bottom of the page aren't counted if the user never scrolls them into view). With the incentive for pagination gone, look for more long, scrolling articles.

A Very Painful Bug

2012-04-01T18:24:00-07:00

I'll lead off this post by listing all of the various things I thought might have caused this bug, and related phrases that I Googled in the hopes that it will lead some poor soul chasing the same issue to this post, thus shaving 2-3 points off that person's long-term systolic blood pressure.

Facebook authentication mobile safari crash
jQuery mobile Facebook crash mobile safari
Mobile safari crash switching windows
mobile firebug
Mobile safari ios crash logs
Understanding ios crash logs
ios crash webthread javascript
juggernaut crash mobile safari
socket.io mobile safari crash

Basically, there is a problem with how Safari handles websocket connections. It manifests clearly when, on a mobile device, a page spawns a child window for something like Facebook authentication (or any Oauth flow). When the browser returns to the parent window after authenticating, Safari will completely crash - or, at least, most of the time it will completely crash. In this case there was a frustrating dependency that made the bug not 100% reproducible.

After hours of debugging (you know you're getting desperate when you resort to binary-searching for the bug by systematically commenting out half your JavaScript...) and Googling, I found this comment on github which describes the symptoms perfectly.

I was experiencing it specifically with a connection spun up by Juggernaut, which is based on socket.io. The solution in my case was simple - just prior to making the FB.login() authentication call, I explicitly close Juggernaut's websocket and unsubscribe the client from my PubSub channels:

jug.io.socket.disconnect();
jug.unsubscribe("myChannel");

And then reconnect everything when I return from the Facebook auth call:

jug.io.socket.connect();
jug.subscribe("myChannel", callBackFn);

Works like a charm, though good god it was painful to identify.

Juggernaut in Windows

2012-03-09T19:49:00-08:00

Note: This is an old post, and no longer relevant as Juggernaut is no longer a thing. Just use socket.io directly.

I've fallen in love with Juggernaut.

Juggernaut gives you a realtime connection between your servers and client browsers. You can literally push data to clients using your web application, which lets you do awesome things like multiplayer gaming, chat, group collaboration and more.

It's built on node.js, socket.io, and Redis and makes implementing PubSub in your web apps super easy. (How easy is it?) This easy:

Client side (subscriber)

var jug = new Juggernaut;
jug.subscribe("channel1", function(data){
    alert("New message: " + data);
});

Server side (publisher - I'll use Python, but Ruby is similar)

jug = Juggernaut()
jug.publish('channel', {'message': 'Hello World!'})

I use Juggernaut to implement real time features in my Flask application. Flask is a traditional web framework and not very well suited to real-time. Juggernaut is a great complement. My development environment is Windows, so I needed to get Juggernaut running there. It wasn't terribly difficult but there were enough gotchas that I thought I'd share how it's done (and that it can be done).

This assumes you have Python 2.7 installed. I would not recommend deploying Juggernaut in production on Windows. Here we go:

Get Redis
- Redis isn't supported on Windows but there is a great native port available that seems to work well enough. I wouldn't run it in production, but it's been stable for me in development. Download it here and extract to a folder anywhere you please.
Get Node.js
- Node now supports Windows! Download and install it here.
Get Juggernaut
- You'll install Juggernaut via the Node Package Manager (npm). Npm is installed by default with Node. Open a command window and navigate to the install directory for Node (this is C:\Program Files (x86)\nodejs on my machine). Run npm install -g juggernaut
Get the Python library
- Armin Ronacher has written a simple Python library to connect to Juggernaut. Install it by running pip install juggernaut at the Windows command line. (don't have pip?)
Run it!
- Fire up redis-server.exe from the directory that you extracted Redis to.
- Run juggernaut from a Windows command line (just type juggernaut)
- Juggernaut helpfully comes with a nice little example implementation (index.html). Npm installs it here by default: C:\Users\AppData\Roaming\npm\node\_modules\juggernaut\public
- Not-so-helpfully, the test implementation connects to Juggernaut over port 80, which is silly because Juggernaut serves messages over port 8080. So crack open that index.html file and change line 37 from "80" to "8080"
- Next, copy index.html and application.js (which is the client side Juggernaut library) to a directory in IIS so you can hit it locally. You do need to serve the file from a web server - you can't just open the static file in a browser because when the sample tries to connect to Juggernaut the browser will view it as a cross-domain request and reject it.
- Now, browse to http://127.0.0.1/index.html:
- Almost done! Finally, fire up IDLE (or a python interpreter) and try the following. You'll see the message immediately in the client app. Cool!

Hope this helped some folks. Enjoy your shiny new real-time web component!

Communicating Performance Thresholds

2012-03-01T22:52:00-08:00

I think visually, so here's a way I like to communicate performance thresholds. There are a bunch of dimensions that might make an application behave differently. Test against those dimensions and plot the thresholds of each one. Here are some that I made up, for a theoretical application.

The goal is to have all real world scenarios fit inside your polygon. So here's a real world use of the application that we can handle no problemo.

And here's one that pushing the limits a bit more.

And here's one we didn't anticipate, or our app should never have allowed because it's clearly not going to work.

There are a few caveats to visualizing performance thresholds this way:

We might have missed a dimension. Turns out that the phase of the moon dramatically affects performance and that wasn't a dimension we tested along. Oops. This is no fault of the visualization, but can most definitely happen (and in fact did just happen and prompted this post).
Dimensions might not be independent. For instance, if you have more memory, the maximum allowable job size may go way up. That's hard to capture here, and often isn't necessary. If you're giving guidance to a customer on performance, skip the nuance and stick to what works.
We might not really understand some of the boundaries as well as this diagram indicates. With performance testing, it's rare that you can categorically say that 1,000 visitors works fine but 1,001 visitors brings down your website. Build in some buffer when communicating.
And remember, there is a difference between an app theoretically supporting a threshold and actually testing to that threshold. Theoretical boundaries need be called out clearly and separately.

Building Software and Building Bridges

2012-02-21T14:57:00-08:00

We have a problem. People can't get from one area of town to a neighboring area because there is a river in between and no road. So let's build a bridge.

Step 1: Requirements

We'll get detailed requirements from civil engineers and government officials, including environmental constraints, safety guidelines, traffic considerations, and all sorts of other arcana.

Step 2: Design

We'll have architects, designers, and engineers create a plan that meets all of the criteria. We'll do a tremendous amount of pre-planning. We'll take public opinion surveys. We'll invest a huge amount of time and energy up front to ensure that everyone knows exactly what kind of bridge we're building and exactly how much it will cost. Our engineers and designers will have estimates of exactly how many pounds of concrete we'll need. And how many nails. And which suppliers the nails will be coming from. And backup nail suppliers in case something happens to the primary supplier.

Step 3: Implementation

Contractors go to work. Using the detailed design a team of professionals can reasonably be expected to deliver this massive project on time, and on budget. And we have our bridge!

Hmmm...

Requirements--> Design--> Implementation

Does that sound familiar? Oh yeah! It's waterfall development! And it doesn't work for software.

Since the beginning, we've tried to wrap traditional project management and construction metaphors around software development. It sounds reasonable - in both cases we have teams of designers and engineers delivering a very complex, expensive solution over a long period of time. In the construction world, if we're over budget or late, it's probably due to some flaw in the requirements or design process. If only we had invested more time up front! As software projects blow past their budgets and deadlines (whoosh!), companies also often react by investing more up front. We get long, detailed requirements documents that no one reads. Using construction techniques to solve software problems doesn't work because it's a flawed analogy.

Let's try that bridge again, but with software

We're going to start by focusing on the problem to solve: get people from A to B. With software, the solution isn't necessarily as obvious as it is in the physical world. Maybe we need a bridge. But maybe we need a ferry. Or a helicopter service. Or maybe we should just move the two pieces of land closer together. Or freeze the river.

Customers speak in terms of solutions: I want a bridge. I want a bigger kitchen. But with software we know to be wary of this: unlike the physical world, the users of software often do not have a good intuitive understanding of what's possible. So while they speak in terms of functionality and solutions, it's our job to root out the real problem and come up with an appropriate solution.

But let's say a bridge is the right answer and we've settled on the bridge we want to build. It's big, it's complicated. Unlike building a real bridge, spending lots and lots of time up front on our software bridge will not lead to accurate estimates for our bridge's construction. There are lots of reasons for this: programming is fundamentally an individual task and developers work at dramatically different levels of productivity. We may not know which developers are working on our bridge. Or the ones that provide the estimate may not deliver the parts they estimated.

Furthermore, software complexity increases exponentially with the size of the project. A 10,000 line program is hundreds of times more complex than a 1,000 line program. But a 10,000sqft house is linearly more complex than a 1,000sqft house*.

The result is we have a unique statistical challenge when estimating software. Michael Church on Hacker News explains it beautifully:

Let's say that you have 20 tasks. Each involves rolling a 10-sided die. If it's a 1 through 8, wait that number of minutes. If it's a 9, wait 15 minutes. If it's a 10, wait an hour.

How long is this string of tasks going to take? Summing the median time expectancy, we get a sum 110 minutes, because the median time for a task is 5.5 minutes. The actual expected time to completion is 222 minutes, with 5+ hours not being unreasonable if one rolls a lot of 9's and 10's.

This is an obvious example where summing the median expected time for the tasks is ridiculous, but it's exactly what people do when they compute time estimates, even though the reality on the field is that the time-cost distribution has a lot more weight on the right. (That is, it's more common for a "6-month" project to take 8 months than 4. In statistics-wonk terms, the distribution is "log-normal".)

Software estimates are generally computed (implicitly) by summing the good-case (25th to 50th percentile) times-to-completion, assuming perfect parallelism with no communication overhead, and with a tendency for unexpected tasks, undocumented responsibilities, and bugs to be overlooked outright.

Software teams will not be able to tell you the number of nails needed. Or the pounds of concrete. Software estimates are unintuitive and must account for a great degree of uncertainty.

Good software teams use a completely different set of techniques to battle uncertainty: They ship frequently (there are not many shippable increments when building a bridge), run lots of experiments, and integrate the design, implementation, validation, and requirements phases together into one process. We encourage iterative, creative thinking. We want to try lots of things, fail fast, and move on. It's not unusual to read something like this about building software (from Little Bets):

Learn by doing. Fail quickly to learn fast. Develop experiments and prototypes to gather insights, identify problems, and build up creative ideas, like Beethoven did to in order to discover new musical styles and forms.

It's not surprising that the waterfall approach to software has failed. Waterfall was taken from the world of construction, but construction is simply not analogous to software. We need new processes.

It's only in the last ten years with the emergence of Agile, Kanban, rapid prototyping, and general acceptance of iterative, fail-fast approaches that software development has come into its own. We are good at building bridges, and we're finally getting good at building software.

* I was a little uncomfortable making this assertion without knowing much about construction, so I called up a friend who is a project manager for medium/large sized construction projects (industrial parks, housing developments, etc) and discussed. He backed the claim.

No Bugs != Quality

2012-01-29T19:48:00-08:00

A low bug count is not a good indicator of quality software. Lack of typos and grammatical errors is not indicative of a quality book.

If your software has nagging "quality problems", driving the open bug count to zero probably won't be much help (not least because it only addresses the bugs you've found - not the multitude inevitably lurking in obscure code paths). If the software is hard to use; if it doesn't solve the problem it was designed to solve; if it's slow; if you pay a huge engineering tax whenever you make a change because the code base is spaghetti; then bug count may not be your problem.

In fact, a high bug count isn't even a reliable indicator of poor quality. Google Chrome, the world's second most popular browser, has over 32 thousand open issues.

I'm all for keeping bug counts low - fewer bugs is better than more bugs - but if you have a large number of infrequently-reported bugs, then bug count is little more than a vanity metric.

Fixing every outstanding edge-case bugs will not create happy users out of net detractors. Playing whac-a-mole with bugs is ineffective. Quality issues demand stepping back and taking different approaches. Bugs are symptoms, not causes.

Localization (or: Localisation) Tip

2012-01-03T21:22:00-08:00

I just learned something from our documentation team: Try to avoid using gerunds in UI text because they are difficult to localize & translate. While they certainly aren't a challenge for great translators, I'm told they can be misinterpreted by mediocre translators - like the ones that outsourced translating services might use.

"Creating a page" --> "How to create a page" "Click here for help resolving this issue" --> "Click here for help with this issue"

How to Improve Chrome Web App Adoption

2012-01-03T17:51:00-08:00

There was big news in the browser world this month, as Google's Chrome browser eclipsed Firefox's market share, making is the #2 most popular browser in the world. Google's goals with regard to Chrome have always been clear. As Peter Kasting, a founding member of the Chrome team at Google, puts it:

"The primary goal of Chrome is to make the web advance as much and as quickly as possible"

While I don't doubt this as the primary motivation for building the browser, there are clearly some tangential opportunities that Google is exploring. For instance, there is a fascinating theory that the Googlebot (the crawler responsible for maintaining Google's search index) is in fact running Chrome under the hood in order to index JavaScript and other dynamic content.

There is also the Chrome Web Store, which has emerged in over the last 12 months as a way for Chrome users to browser through a (very) lightly curated selection of web "apps" and install them to Chrome - an app store in the footsteps of Apple, for the browser & web. The store appears to have about 65 million users (though it's unclear what counts as a user) and about 18,000 apps available.

I have some theories about Google's strategic goals of the Chrome store (in all likelihood it's to support Chrome OS, but it might not be. What else, say you? To drive Chrome adoption? To turn a profit by pushing Google Checkout? To simply "advance the web"?) but ultimately we can make the safe assumption that the success of the Chrome Web Store is predicated on people installing and using Chrome apps. More app installs = execution of strategy (whatever it is).

So how is the store doing? With 65 million users, based on Chrome's \~25% market share and the 2bn worldwide internet users, about 13% of Chrome users have made use of the store. Given how the store works today I'm actually surprised it's that high, but I'm guessing that iPhone users' adoption of the Apple app store is close to 100% and that the Google team is not satisfying mucking around at 13% adoption.

Let's see if we can help them out.

But first, it's worth noting that there are really easy ways to drive Chrome app adoption: aggressively promote the store from within Chrome itself, show app results in Google search results, and in general promote the store across Google properties. However Google has wisely resisted such cheap promotion in the past (edit: maybe not), so this discussion is focused on how the Chrome store can succeed on its own merits, not on Google marketing it to a captive audience.

So let's try and divine what exactly a "Chrome App" is, versus a "normal" web app that's just run through the browser. I'll go ahead and install the ever-present Angry Birds (which over 1m Chrome users have also done). It's easy enough through the Chrome Store, and here's the end result:

I installed Angry Birds!

Voilà! I now have an Angry Birds icon available whenever I open a new Chrome tab.

When I click it, Chrome navigates to http://chrome.angrybirds.com where I can, surprise, surprise, play Angry Birds. It's worth noting that I can just as easily hit this URL without the Web App installed. In fact, I can even hit in IE and it works fine.

So what's the big deal? Surely there is something to Chrome Web Apps other than a handy link on new browser tabs, right? Well, there is. The difference is near the bottom of the screen, and here's what it says when I have the app installed:

Ah-ha! It's installing the offline version. And indeed, without the app installed, I see this instead:

In fact, it's the same message I see in IE:

Hmmm, well that seems...minor. The handy shortcut is fine and good, and apparently the app also gets access to some local storage, but if that's it then the store may be in for a lot of comments like this, from a Google's "Why You Should Create an App" video:

But Angry Birds is just one example - there must be more, right? Let's hear from Google directly, which will surely convince us that the app store is more than a directory of useful web sites.

Here is how Google describes the benefits of a web app, buried deep in an FAQ:

Why would someone install a web app instead of just using a bookmark or typing in a URL? Installing an app ensures it is launchable from the New Tab Page via the Chrome apps launcher. Installing an app also allows you to grant it privileges such as unlimited local storage and background pages.

That's pretty much the entirety of Google's explanation of why Chrome apps exist. Even the way the question is posed makes the apps sound ridiculous.

To encourage app installs, the benefits of doing so need to be obvious and tangible. Right now they are neither. There are in fact real benefits to the app approach, but Google has seemingly little interest in conveying them to end users. As a result, users view apps as glorified bookmarks.

The situation for developers is no better. The developer portal describes the problem apps solve as "requests for permission are annoying" (really, this is what it says). Again, benefits are not spelled out and developers are left guessing as to why, exactly, they would make an app.

Ask an iOS developer what the benefits of a native app are (even shimmed HTML5 apps made with technologies like PhoneGap). They'll say:

Frictionless purchasing
In-app payments
Rich APIs
Exposure via the App Store
Available offline/no network lag
(and yes) Elevated permissions

In fact, Chrome Apps offer many of the same benefits but they are poorly explained to both users and developers. Paid apps, integrated with Google Wallet, are available (albeit poorly adopted - this was literally the only example I could find after 20 minutes of searching), as is an in-app payments API.

Google just needs to explain this better. And it's not hard. Here's 10 minutes of effort in Balsamiq:

Google needs to make the benefits clearer.

But even existing apps are not taking advantage of these benefits. For instance, Prezi is a fairly popular productivity app in the store, with about 50,000 users. As far as I can tell the app is literally a glorified bookmark, just linking to Prezi.com. I still have to sign up (create a username and password) for Prezi.com, and if I want to purchase a paid version I get a form that looks like this:

Wowza! Lots of form fields! Given that I have installed this as a Chrome app, shouldn't Prezi know something about who I am, and have some sort of access to my data? At best, signing up should be a one-click process (similar to purchasing an app in the first place) and at worst, couldn't it at least have an account already created for me and pre-populate some of the fields? It's unclear to me how much of this is possible with the current APIs, but this type of thing should be the default, expected behavior for anything installed as an application (and only applications that take advantage of the benefits of being an app should be featured in the store). Rapid, secure registration and payment are clear benefits to both users and developers of apps.

If I haven't installed Prezi as an app, Google should allow developers to easily incorporate the "Add to Chrome" button to their site, which would allow the site to then take advantage of easy registration and payment options. Imagine if, above all of those Prezi registration fields, there was a button:

A user could then choose between simply clicking this button, or filling out then entire form. I'd click the button every time for rapid, safe, and centralized payment and registration.

Finally, the store itself needs better curation. Browsing around the Games category, it seems half of the games are simply links to little Flash games surrounded by ads - hardly the slick HTML5 "apps" Google wants to promote. With a higher threshold for quality, and some improved discovery features, the store could quickly become a great way to find new services.

In conclusion, the Chrome store has great potential but needs to make changes to avoid becoming nothing more than a cool bookmarking service. Specifically:

Make the benefits of apps clearer to users and developers
Create better registration and payment APIs, and help developers incorporate them into existing services
Create an Add to Chrome button that can be used outside of the store itself
Improve quality and discoverability of apps in the store

I look forward to see how store adoption grows, and how this new take on the app concept plays into Google's larger strategy and offerings.

DOET Shower

2012-01-02T15:12:00-08:00

I stayed at a friend's apartment a few weeks ago. It is difficult to describe the decor, but if my grandmother and Master P were forced to collaboratively design an apartment, his place is what you might get.

Walls covered in paint made from melted silver, gold dragon faucet heads roaring out of the wet-bar sink, all trimmed with flower-print or toile curtains. That sort of thing.

The shower is, of course, also over-the-top; one of those awesome luxury showers with heads and nozzles jutting out every which-way from the granite walls. Sounds like great fun, but when it came time to actually use it, I was confronted with the following:

Not only did I fail to figure out how to control the temperature and which shower heads were active, but I couldn't even figure out how to turn the thing on and off! I was assaulted from all sides by water of all different temperatures during the minutes of fiddling it took for me to turn this stupid thing off.

I was horrified enough by this set of controls that I actually bothered to take this photo to document the insanity.

Yesterday I started reading The Design of Everyday Things by Don Norman and, to my delight, came upon the following diagram:

With the wonderful caption: "What do you do here? Why would anyone dream up this scheme?"

I am very much looking forward to reading this book.