Skip to content

The Big Tech Doc

Jump to bottom
Claire Annan edited this page Feb 19, 2023 · 8 revisions

Do you prefer to dive right into the code instead? Please read the README for instructions to run the prototype locally.

Table of contents:

Background and Goals

This document is for engineers new to the Guides and Methods Team. It also benefits anyone that wants to learn about Hugo as a static site generator, why it’s a good option for our partners, and best practices for building software for content editors.

If you prefer to just get started and explore on your own, follow the repository README for local development instructions. At the time of this writing, there are 10 guides in 10 separate repositories. The Guides and Methods team created a prototype to demonstrate how to centralize the guides into one repository while improving performance and maintenance.

Following the 18F model, we used this opportunity to address the content across the guides. This prototype demonstrates how a new, centralized information architecture could be implemented.

We had a lot of technical hypotheses, which this prototype can continue to test: Airtable of hypotheses with the Technical label. Full list of architectural hypotheses. For a list of the existing Guides and their GitHub repositories, see the 18F guides index (18F Only 🔒)

Why Hugo?

The Guides and Methods team needed a JAMstack git-based framework that can run on cloud.gov Pages to replace Jekyll. It has to be easy to learn, fast for prototyping, easy for vendor developers to pick up and manage in sandboxes and in production. We identified common developer tasks and examined how quickly and easily different frameworks allowed us to implement them. Hugo stood out by having excellent documentation and many pre-built (out-of-the-box) functions to perform these tasks. See Benefits.

A collection of Technical Requirements for the Guides & Methods: Technical Requirements

A presentation on the decision process: Engineering Huddle - Replacing Jekyll for our Guides & MethodsEngineering Huddle - Replacing Jekyll for our Guides & Methods

Benefits

You don’t have to know Go. Although Hugo is written in the Go programming language to support multiple platforms, you don’t have to be a Go developer. You don’t even have to install Go. Just grab a precompiled binary from Hugo’s Releases page. [Explain how this works on Pages]

Git-based. Independent of Git version control provider (GitHub, which is low FISMA, GitLab which is moderate FISMA)

Easy to follow documentation. I installed Go locally on a Mac with Homebrew, following Hugo’s Getting Started guide. Within minutes I had a working local site. It took me __ days from initially poking around the documentation to having a fully fledged prototype. See snapshots of my progress in the GitHub repository.

Standardized functions to create new content using archetypes. Archetypes are content template files in the archetypes directory that contain preconfigured front matter and even content disposition for the Guides and Methods content types. These are used when you run the hugo new command. See how we implemented archetypes in Creating new pages and Creating new sections. Having archetypes allows us to potentially connect these functions to content management system actions.

Live reload by default allows us to make the changes to the site configuration, (or any other file in the Guides and Methods site) while the Hugo server is running, and instantly see changes in the browser. This means we don’t have to stop/start the server to pick up config changes. You’ll see the log message Change of config file detected, rebuilding site. You might have to hard reload your browser cache once in a while.

Git submodule support. Use these to add themes without worrying about maintaining them. Updating the theme is as simple as running a git command, instead of manually updating files.

Template partials can be used for USWDS components. This allows us to update USWDS HTML separately.

Simple multi language support. Jekyll notoriously made multi language support hard and had no out of the box options. Various plugins had to be installed and not all of them were maintained. Hugo offers not only makes setting up multi language easy on Guides and Methods, it handles URL permalinks and redirects flawlessly and has options to spin up servers per language. Since all URLs are generated from the root, the English home page .Permalink is set to https://example.com/, not https://example.com/en. If a file has no language code, it will be assigned the default language of English.

Page resource handling allows images, pdfs, and other documents, can have page-relative URLs and their own metadata. Page resources are only accessible from page bundles, the directories with index.md or _index.md files at their root. Page resources are only available to the page with which they are bundled. You can create page bundle archetypes, which is a superb feature! It allows us to create entire sections with default content just by running a script. Imagine adding an entirely new guide with preloaded content, pages, and navigation just by running a one-line script. Not only is this a time saver for engineers, but this script can be tied to a content management system action. Potentially, a product owner could start a new guide with no engineering work. Read about Creating new sections to learn how we implemented this.

Dynamic menus. Hugo makes no assumptions about how your rendered HTML will be structured. Instead, it provides all of the functions you will need to be able to build your menu however you want.

Section page templates are a HUGE asset. They allow us to create custom layouts for a specific section. Just by giving the custom layout the same name as a section, Hugo automatically recognizes it and will apply the customized template layouts instead of the parent theme layouts. This is a powerful tool to create a section that has a different “look and feel” but maintains the same infrastructure behind the scenes. See the “fancy theme” description to see how we implemented this.

Blocks allow you to override parts of the base template (for example, a footer) in other templates. This feature makes templates more powerful and layouts more modular. **Goldmark **is the default Markdown parser as of Hugo 0.60. Hugo creator Bjørn Erik Pedersen (also known as @bep) says it’s “fast, it’s CommonMark compliant and it’s very flexible.”

Cached partials for header and footer Partial Templates | Hugo

Google Analytics. Provide your tracking ID in your configuration file. Also Internal Templates | Hugo

Fingerprinting for any asset. The default hash function is sha256. Other available functions are sha384 (from Hugo 0.55), sha512 and md5.

Other cool benefits

  • Image minification and resizing
  • Simple sitemap generation

Watch out for these gotchas

Read how we did this in the ‘fancy’ layout template.

Getting access

Hugo repository

This repository is public and visible to anyone, per 18F’s open source policy. To grant someone access, go to Settings > Collaborators and teams. After multi-factor authentication, you’ll see the list of groups and people with access. We created the guide-admins group for members with an Admin role. You can also add members with more limited Read and Write roles.

Pages build history and logs

The Guides and Methods Hugo site runs on cloud.gov Pages. To access see build history and logs, visit your Pages dashboard at https://pages.cloud.gov/sites.

  1. Click Add site.
  2. Paste https://github.com/18F/guides-and-methods-hugo as the repository URL.
  3. Select 18F as the organization.
  4. Click Add repository-based site.

Once the site is successfully added, you should see 18f/guides-and-methods-hugo in your site list. Click the title to view the site build history for every branch. Build logs will only be available for 180 days after the build completes.

If you need access to cloud.gov Pages, email [pages-support@cloud.gov](mailto:pages-support@cloud.gov). You can also ask for help in the Slack #pages-support channel or fill out the [Pages Contact](https://federalist.18f.gov/contact/) form. Make sure you provide your GitHub username and our repository URL.

Pages site previews

Every branch is built by Pages with a site preview. The preview link is auto-generated within a [pull request](https://github.com/18F/guides-and-methods-hugo/pull/29), also known as a PR. At the bottom of a PR, click on Show all checks > pages/build > Details. Note that previews are only available after a complete build, which is super fast with Hugo. Share these URLs when you’re asking for feedback or approval on a change. It’s good practice to include the site preview link in your PR. Your reviewer does not need a GitHub account to view a preview link. [screenshot of How to get a Federalist preview URL] You can also find links to site previews under each branch name after logging into Pages.

Analytics

Since the prototype is not live, Google Analytics was not implemented. To access analytics for the existing Guides using the Digital Analytics Program (DAP) dashboard, visit https://analytics.usa.gov/general-services-administration/data/. You’ll have to download the csv or json file and manually search for 18f.gov domains. For targeted data with real-time reports, visit the Google Analytics dashboard. Ask Julie Strothman - QUEACG to add you to the 18F Analytics Account. You’ll find views for each Guide under the Microsites (Pages) property. Read more about How analytics work for the Guides doc. Watch a walkthrough of analytics recording.

Search.gov results

Since the prototype is not live, Search.gov was not implemented, but should be before it launches. To access search.gov results for the existing Guides, email [search@gsa.gov](mailto:search@gsa.gov) with the URL of sites you’d like to review. Once you have access, log into https://search.usa.gov/ and select a site. You can generate reports based on queries, clicks, and referrals.

Touchpoints

Currently, only the existing Methods site uses [Touchpoints](https://touchpoints.app.cloud.gov/touchpoints/d028f982/submit).

To access the Touchpoints form and results, please contact Julie Strothman - QUEACG, Anne Petersen - QUEACA, or Melissa Braxton - QUEACI. Once you get access to Touchpoints, you must log in once every 90 days or your account will be deactivated, requiring you to email Touchpoints staff for reactivation. These deactivations happen at the end of each month.

Keeping the repository healthy

Dependencies As a site administrator, you should receive email notifications for Dependabot alerts. Dependabot will create a new branch and a pull request but you have to merge it into the main branch. Regularly merging in dependabot updates not only keeps you up to date with dependencies and security. This activity prevents your repository from being auto-archived or “frozen”. Plus, having outstanding pull requests is just a bad look since these are public repositories. Branch protection Make sure you enable branch protection to the main branch. Go to Settings > Branches and apply the following options: Require a pull request before merging Require approvals (at least one) Require status checks to pass before merging Currently Build deployment – pages/build (cloud.gov) is the only check set on the main branch. See Future work for next steps.

Branch protection Settings > Branches Apply to the main branch. Branch protection rules > Require branches to be up to date before merging: pages/build Once this setting is updated, you’ll the the build status under “View all checks”

Check passed confirmation screen Auto-archive prevention This repository includes the topic maintained to prevent auto-archival. [screenshot of repository details] Webhooks If you’re an organization manager, you can add users and assign them roles of manager or user. You’ll need their id (for example, mine is 902). Pages also provides sandboxes as their own organization. These sandboxes are useful as a staging environment outside of the public 18F organization. This is especially helpful when testing out workflows that send out notifications. Go to Settings > webhooks Click Add webhook Enter Payload URL: https://pages.cloud.gov/webhook/github Select application/json as the Content Type Cloud.gov Pages team will provide you with Secret Keep the rest of default settings and click Add webhook to save changes If successfully added, you’ll notice a check icon in front of the webhook with the label “Last delivery was successful”

Delete stale branches After a successful merge, delete the branch. Running accessibility checks Run the following command locally. Hugo’s default port is 1313.

pa11y-ci -s http://localhost:1313/en/sitemap.xml --sitemap-find "//localhost:[PORT]" --sitemap-replace "http://localhost:[PORT]/"

Pa11y should be added to the CI/CD pipeline. Currently we’re using GitHub Actions. Axe or htmlcodesniffer (add version numbers here) Checking performance During the initial build, you’ll see a summary of total assets, build and paint time. [Screenshot] You can run the following commands for more details Hugo also offers configuration options to improve performance. Link.

A tour of the Guides theme

Components: themes/guides/layouts/partials, themes/guides/layouts/partials/uswds Layouts: themes/guides/layouts USWDS updates The prototype uses USWDS v 3.? Types of layouts homepage list basic fancy sitemap Page options (also known as front matter): formats and defaults Link title: string Hero: false Breadcrumb: true Side navigation: true In-page Navigation: true Wordcount: # words ReadingTime: # minutes read Aliases: array BundleType: Publish date: today Last modified: January 1, 0001 Sort weight: 0 Is translated: false Previous page: Page With Code Next page: In-Page Navigation Page Tags: accessibility methods Summary: Components You can find custom and USWDS components in the > > > directory. We separated custom components from USWDS components intentionally. They are named just as they are in USWDS documentation. Since there is yet no push/pull mechanism for updating HTML from USWDS, this practice allows us to keep our HTML clean and reuse components. Example: identifier.html How the navigation works Menus | Hugo - here’s how navigation works in Hugo in more detail. In general, Hugo will generate certain lists based on the type of page you’re creating: the first page in a section will list all the pages within that section, which is powerful. Hugo’s menu creation abstracts relationships between content no matter what that content is. Independent of the type of content that you have, Hugo has functions to recursively examine that content and generate things like the page title, URL, author, front matter, and metadata from the content there. Hugo has menu templates, or you can create your own. For a breadcrumb template — what we call side navigation is a list of sections and the pages within them (recursive). If you want to create your own, Hugo has functions to do so. For the Guides + Methods, we use a combination of menu templates and custom menus. You can find those under themes -> guides -> layouts -> partials and themes -> guides -> layouts -> partials -> uswds A common engineering task is displaying as a list all page titles of nested directories of a current section. Jekyll and 11ty don't have functions to do that. [Show example of how this was done on other sites]. Hugo does.

Pushing changes to production

Where the content is In this GitHub repository, find content organized by Section under content/en, or use the GitHub search feature to find the file. All content files end in .md for Markdown. Hugo also supports Emacs Org-Mode, AsciiDoc, RST and Pandoc content formats. Two ways to update: either an editor like VS Code (engineers usually use this) or the GitHub web interface (content managers often use this). Pages that can be managed by content managers are intentionally separated from what engineers may need to change. From the root it would be the content directory. For this prototype, multi-language support is enabled: en and es (English and Spanish) folders. Within that directory, it’s in the structure of the information architecture. Section 1, 2, for right now. [Screenshot directory of the folders] Here you can see the root (en) versus the sections. Hugo recognizes any section within the content folder. Hugo will automatically create a page for each root section that lists all the content in that section. For the Guides and Methods site, we name this list page _index.md. These high-level folders are Sections: this is a great feature in Hugo that lets you create navigation just due to the folder structure indicating portions of the site. (11ty and Jekyll do not provide this; engineers usually have to build this _index and homepage of the subsection, and how to get the title or metadata from that page.) Hugo does this via a function. Documentation here: Content Sections | Hugo. Hugo provides variables like .Section, .IsSection, .CurrentSection, .FirstSection which are incredibly useful to create dynamic navigation systems. See how we implemented this in the How the navigation works section.

Updating content from the GitHub UI

From the main GitHub repository, navigate to the content directory / folder, then en (English) folder, then the section, [then subsection if applicable], then down to the page you’re looking for. Click on the edit pencil in top right of the document / page content. [screenshot] At the top of the document between the three dashes at the top of the page and about six lines down, is the front matter where we see our page options, including Title. The options are Boolean values, so turn them on or off using True or False there. Changing the title here will show up throughout the site even if the different types of navigation like the main navigation or side navigation: the change once, change everywhere principle that’s so useful in Hugo. After the front matter dashes, the actual content appears, which is all in Markdown. After making changes, scroll down to the bottom to the Commit changes box. Give it a meaningful commit message to show what you’ve done. Then select Create a new branch for this commit and start a pull request, and leave your name in the branch name, but rename the end from patch-1 to a meaningful description, using dashes rather than underscores between words. This will create a new branch after you click Propose Changes. On the next screen, it will say “Open a pull request” at the top. This should contain an issue template, prepopulating the comment section with instructions about what to include (often asking for things like what was changed and why, preview URL which you may leave blank for now, or a few other items). Clicking Create pull request will generate a preview URL. Select who to review on the right: the current team should have a standard about who to add. After clicking Create pull request, you’ll see the page for the Pull request. Scroll down to make sure that all checks have passed. Click on Show all checks: you’ll see cloud.gov Pages built, saying build is complete, and for the Preview URL, click Details to see the version of the page with what you changed in its new branch. Check your work, then copy the URL of that page, and go back one page to the Pull request itself, and edit your pull request (top right above your message) to add the Preview URL to the body of the pull request itself. When clicking into the Pull request tab at the top, you can then see the list of Pull requests which should now include the one you just created. It will be waiting for approval until someone reviews it and merges it into the main branch. At least one approving review is required, and all checks must pass (though admins may be able to bypass this —though typically shouldn’t). Only after that will it be viewable on the main site. The Check process now uses GitHub actions, but in the future may decide on using CircleCI. Security, dependencies, accessibility should fall under the Checks in all Pull requests. The Preview URL won’t be generated until the last check completes, which is all part of the build. Hugo typically has seconds of build time rather than minutes. Make content changes. Attach files by dragging and dropping, selecting or pasting them. Save your changes by clicking the Commit changes button. This will create a new branch for this commit to start a pull request. The guides-admins group is set as the default reviewers. Preview your changes using the cloud.gov Pages preview link. [front matter: turning on and off options. Mentioned earlier (link here)]

Reviewing pull requests Go to the pull request page, click on the Preview URL to review and/or switch to the Files changed tab to look at the Markdown that has changed and make sure there are no mistakes or changes you would recommend. If there are, the reviewer may make the change, the reviewer may just use a GitHub comment about what the change should be, the reviewer may Slack the originator to talk about it (or otherwise discuss), or otherwise ask the originator to make the change. If no changes are necessary, you can just click Approve and add a comment to that effect: often “LGTM” for “looks good to me.” After you have done a review, added your comment and clicked Approve, the next step would be to Merge that branch. Each team may decide differently if they want the requester or the approver to make the changes and/or do the merging which will make that change live. That should be a governance decision and done in a standard way. We recommend the reviewer do the merging and deleting of the branch. In this repository, you can’t Merge without checks passing — this is a setting at the repo level.

[Comment Submit general feedback without explicit approval.

Approve Submit feedback and approve merging these changes.

Request changes Submit feedback that must be addressed before merging.]

Expect to see: [​​Changes approved 1 approving review by reviewers with write access. 1 approval All checks have passed 1 successful check This branch has no conflicts with the base branch Merging can be performed automatically.] If there’s a drop-down, the default “Create a merge commit” is what you want: just don’t click the dropdown arrow and use Merge pull request. The next message will default to the proposed change text: you can just leave that there and click Confirm merge. A successful result will show “Pull request successfully merged and closed You’re all set—the ‘new’ branch can be safely deleted.” and you’ll get an email confirming. Delete your branch once it has been successfully merged. (part of Keeping your repository healthy) This is the responsibility of the person doing the merge. After doing that, you should see “deleted the ‘new’ branch now”

Creating new pages Until we implement a content management system, creating new pages manually. Using the GitHub UI, navigate to section within the content folder. click Add new to create a new markdown page. Add the following as front matter:

---
title: "New page"
breadcrumb: true
inpagenav: true
sidenav: true
categories: "Buy better"
tags: [accessibility, methods]
---

We also created a default archetype to make it easier for engineers to create new pages prepopulated with front matter. Run the following command via the command line from root: hugo new <SECTIONNAME>/<FILENAME>.<FORMAT> The above command invokes the archetypes > default.md template to create a new page with the basic content type, adds a title front matter variable, and sets the title value to a string of text from the file name. For example, if we have want to create a new page called My First Page under Section1, run the following command: hugo new section1/my-first-page.md Creating new sections Currently, we can only create new sections from the command line using the following command from root: hugo new --kind section-bundle <SECTIONNAME>

Clone this wiki locally