The Autogram newsletter intends to cover:
- news about Autogram and what its partners have been up to;
- news about the world we research for our clients: the business and technology of online publishing.
This issue is mostly about the former: a next-generation web crawler, developed by Autogram as an open source project called Spidergram. But, as you’ll see, it’s intimately bound up with the latter, since Spidergram was developed not just to fill a gap in the software landscape, but to help the company’s partners ask better questions to serve our clients.
Autogram frequently works with organizations that have large, complex websites. These sites publish massive amounts of documents, authored by multiple teams, in different formats, and delivered to multiple places on a wide range of devices and platforms. Interlocking architectures, content strategies, and design systems are required to preserve a consistent experience and brand identity while also allowing for enough variation to serve different needs, users, and contexts.
Complex websites lead to correspondingly complex problems. “On one of the spectrum, you have someone who wants your help building a shed in their backyard,” says Autogram’s Karen McGrane. “On the other, someone wants help planning out a city.”
“At a certain size, an organization starts losing track of all the stuff that it has,” adds Jeff Eaton. Even taking a simple inventory of everything published to a network of sites and subsites can be difficult. Getting a meaningful picture of the total architecture, seeing how different parts of the site connect to each other — the roads and bridges of that city — and understanding how users approach that navigation can feel nearly impossible.
All of this affects how an organization’s digital presence is used and perceived. “A lot of the clients we have worked with have developed systems at scale,” says Ethan Marcotte — whether they’re design systems, or content management systems, or other organization-wide technologies. “And it can be hard for them to measure impact, to get a sense of how successful or not those systems are,” or whether they’re used consistently across multiple teams and domains.
Factor in metrics for page performance, accessibility, and readability, and you begin to get a holistic picture of the overall health and usability of a website — not just a spreadsheet of URLs, but an opinionated take on where a site is succeeding and how it might do better.
Existing web crawlers (also called spiders) are typically built for other, narrower purposes: simple inventory tools, search engine optimization, developer-focused “scraping” tools designed to extract highly structured information from specific pages, or ledgers for labor-intensive manual appraisals. Spidergram was built to answer different questions, iterably and at scale. So Autogram decided to build it ourselves.
Jeff, Spidergram’s chief developer, explains that Spidergram brings a wide range of objective metrics into one place, and then allows the team to layer in subjective assessments. This makes answering overlapping questions about complex websites much faster.
Jeff gives this example: “How many things are outdated and are in a particular category? And are getting high traffic?… We can collapse what would have been a week or two of mucking around in spreadsheets trying to match things up into ten minutes now. It doesn’t even feel like automation. It feels like asking a question and getting answers back, seamlessly and at scale.”
This allows our team to ask lots of questions of any website, to work with our clients to answer questions they might have, and to see which answers may match expectations and which might require more digging.
Any organization might benefit from an external health check, but this is particularly useful before undertaking a large project like a replatforming or redesign. This is when web publishers (and at this point, any organization with a sufficiently large online presence is effectively a publisher) need to take stock of what they have and bring various stakeholders together to assess what is and isn’t working.
Karen explains that organizations are tempted to treat a new design system or CMS as a purely technical solution to a technical problem, rather than an opportunity to bring different parts of the organization together to identify and understand specific pain points and common needs.
An inventory of a site’s content and metrics is just the first step in that process. “You're gathering the inventory for the sake of doing an audit on it so that you can make some decisions about what you're going to do,” says Karen. “And you don't actually really want the audit; you want the decisions that come from having done the audit that will then guide future development… You want to make informed recommendations for what should happen in the future.”
“We start a lot of our engagements with a client workshop, whether they’re working on a new design system or a new CMS or revisiting an old one,” says Ethan. “Those are really just sort of opportunities to get different representatives from different disciplines in the same room together. And often, those are the first times those folks are getting together and talking openly about their problems.”
It’s that interdisciplinary, domain-driven, and humane approach to understanding websites and the teams that make them that differentiates both Autogram and Spidergram.
“Autogram's business isn't software development — helping organizations answer pressing questions and make critical decisions is what we're about. But these tools are also an essential part of the process,” says Jeff. “Had another tool already been out there that did what we needed it to do, we would be using it and doing the same work, but just with a different tool. But it didn't exist, so we had to build it.”
Autogram is already using Spidergram to help some of our clients begin new projects. We’re planning on sharing insights from our research on a large government website to showcase Spidergram’s abilities to the public. Because Spidergram is an open source project, users and other developers have been offering feedback. And qualified enterprise clients can request a free Spidergram report by contacting us or subscribing to this newsletter.
Articles and Announcements
The Best Design Systems Are a Shared Work Language, by Lauren Shapiro
From our experience, a design system creates a lasting impact when a shared language — including the vision, mindset, and process — is established from the beginning, then evolved with the input of a cross-disciplinary group of stakeholders.
The WebAIM Million: The 2023 report on the accessibility of the top 1,000,000 home pages
Our 2023 analysis saw small decreases in the number of detected accessibility errors and WCAG conformance failures. Some sectors, such as government and education, fare better than others. While it is good to see progress, significant work remains to be done to make the web accessible to everyone. We at WebAIM hope that this report will help influence improved accessibility.
Content doctrine: Bridging the gap between strategy and tactics, by Jeff Eaton (May 3 @ Confab)
A team’s unique perspectives—its beliefs about how tools should be used and why specific decisions make sense—usually go unarticulated… Those invisible principles are a critical bridge between strategic plans and tactical follow-through. Uncovering, articulating, and documenting them produces doctrine: a shared playbook that can keep your team on track through staff turnover, shifting priorities, and changing technology.