Changelog

When you’re tasked with monitoring performance for numerous websites, it can be challenging to keep up-to-date with the status of each one. It’s even harder to know when to take action.

Because of this, it’s easy to be unaware of performance issues until they become critical. Gathering data can be time-consuming and often requires manual effort.

Today, we’re excited to announce Team Insights, a new monthly high-level report that provides an easy-to-read summary of all Sites tracked in each team workspace.

Each month, you can receive a Team Insights report for all the team workspaces that you’re a member of.

The report includes:

• Straight-forward data: Site’s performance metrics are evaluated and categorised into groups: Regressed, Improved, and Stable.
• Comparison to the previous month: Each Site’s data is compared to the previous month. e.g.: August vs September.
• Metrics at a glance: Each metric is colour-coded to indicate the severity of performance impact and significant changes to metrics are highlighted.
• Chrome User Experience Report (CrUX): Core Web Vitals metrics collected from real-user sessions. (Where available)
• Core Web Vitals Assessment: Clear indication of whether a Site is passing or failing Core Web Vitals.
• Synthetic monitoring summary: See key metrics Largest Contentful Paint, Cumulative Layout Shift, and Total Blocking Time.

With this report, you can quickly see which Sites need attention, and also enjoy confirmation of hard-fought improvements.

Team Insights is available to all Calibre pricing plans and is automatically delivered on the 1st of each month. Team members can opt-out of Team Insights from notification settings if that’s their preference.

The first report will arrive 1st of November, or write to us if you would like early access.

We hope you find Team Insights a valuable addition to your performance monitoring workflow.

We are retiring the First Input Delay (FID) and Time to Interactive (TTI) metrics from Calibre.

Here's what you need to know:

• First Input Delay has been removed from Google's CrUX dataset, Calibre has removed FID from CrUX Dashboard and Pages reports.
• Time to Interactive was removed from Google Lighthouse in version 10. Calibre has removed TTI from Snapshot Overview. TTI will still be collected during synthetic tests, though it is no longer a recommended metric.

Going forward, we recommend using Interaction to Next Paint (INP) as a replacement for First Input Delay and Total Blocking Time (TBT) as a replacement for Time to Interactive.

Today, we're excited to unveil a new Dark UI theme for Calibre, offering you a sleek and comfortable alternative to our standard light interface.

This new theme is designed for those who prefer a darker interface, need increased contrast, or work in low-light environments.

Dark mode will automatically activate if your operating system is set to dark mode.

You can also manually switch between Dark and Light modes from the user menu in the top right corner.

In addition to the new dark mode, there are numerous improvements to address UI interaction bugs, improve text legibility and boost colour contrast for a more polished experience.

These changes include:

• Improved UI spacing to reduce horizontal scrolling on data heavy reports.
• Chart tooltip contrast and legibility improvements.
• Global Site search visual stability and usability enhancements.

There's a lot of small changes and improvements in this release and we hope you enjoy the new dark mode.

If you have any feedback or suggestions on how dark mode could be improved for you, please let us know.

When viewing a Synthetic test, you can now see if a metric has exceeded budget thresholds.

To learn more about budgets, see the Creating a performance budget guide.

As of today, newly created Sites will have 2 Test Profiles: PageSpeed Desktop and PageSpeed Mobile.

PageSpeed Desktop and PageSpeed Mobile Test Profiles run on Calibre infrastructure using the same simulated networking conditions as Google PageSpeed Insights.

Prior to this change, new Sites were created with Test Profiles using runtime network throttling. Simulated throttling comes with the added benefit of tests that complete faster, so you can get results quicker.

The new defaults are designed to provide a test environment that is more consistent with Google’s PageSpeed, while still encompassing the power and flexibility of Calibre's configurable synthetic monitoring platform.

This change will not affect existing Sites, however you can switch between Test Profile devices at any time by navigating to Site Settings → Test Profiles. Runtime network throttling is still available for all Sites.

New Sites created using Node.js API or Command-Line Client (CLI) will use these new default Test Profiles. You can optionally specify Test Profile settings when creating a Site using the API. if that’s more suitable for your workflow.

For more information on Test Profiles, see the Test Profile documentation.

Our Test Agents are now using Lighthouse 12.2.0 to test your Sites. You can find details surrounding the 12.2.0 release in the official Lighthouse changelog.

Our Test Agents are now using Chrome 127 to test your Sites.

On September 1st 2024, synthetic Test Agents will default to dynamic IP addresses. With this change, Test Agents will use a range of IP addresses to access your Sites, rather than a single fixed IP address.

In preparation for this change, you can select fixed IP addresses for each Site from Test Agent settings. Fixed IPs remain available for all Sites, and you can switch between dynamic and fixed IPs at any time.

Fixed IP addresses are helpful if you have a web application firewall, rate limiting, analytics or other security measures that limit traffic from unknown IP addresses.

If you have any questions, or require help in switching existing sites to fixed IPs, please contact support, we’ll be happy to help.

We've added a new metric to CrUX Dashboard and Pages reports: Round Trip Time (RTT).

Round trip time data is newly available to Google's Chrome User Experience Report (CrUX) dataset.

RTT measures the time it takes for a request to travel from browser to server, and back.

Round trip time is a valuable metric to understand latency that customers experience, alongside Time to First Byte.

Using the updated CrUX Dashboard and Pages reports, you can:

• Understand the latency experienced by visitors to your site
• Compare Desktop, Mobile and Tablet round trip time data
• Identify trends in latency over time

At the moment CrUX includes 75th percentile (p75) data for RTT, with histogram data likely available in the future.

Requests initiated by <link rel="preload"> or Early Hints are now displayed with a visual indicator in the Network request table.

Preload and early hints help to improve page performance by hinting to the browser which requests are critical to render primary content quickly.

By carefully employing these techniques you can ensure that essential resources like hero images, CSS or Fonts are prioritised, leading to faster Largest Contentful Paint (LCP) and less layout shifts (CLS).

Using the Network request table, you can now quickly identify these requests and ensure that essential resources are prioritised accordingly.

Read more about optimising critical requests and early hints.

Our Test Agents are now using Chrome 126 and Lighthouse 12.1.0 to test your Sites. You can find details surrounding the 12.0.0 release in the official Lighthouse changelog.

Our Test Agents are now using Chrome 124 and Lighthouse 12.0.0 to test your Sites. You can find details surrounding the 12.0.0 release in the official Lighthouse changelog.

In February, we launched CrUX Dashboard to help you understand how your site performs in the real world.

Today, we're excited to announce the CrUX Pages Leaderboard, a new feature that provides a high-level overview of your Page’s Core Web Vitals performance, based on real-user data from the Chrome User Experience Report (CrUX).

The CrUX Pages Leaderboard is available for all Sites tracked in Calibre.

Now you can:

• Quickly see Core Web Vitals data for all your Pages
• Sort to find the best and worst performing Pages
• Drill down to see more detailed data for each Page
• Filter by form factor (Desktop, Tablet or Phone)
• Observe recent trends for all reported Web Vitals metrics:
  • Largest Contentful Paint (LCP)
  • Cumulative Layout Shift (CLS)
  • Interaction to Next Paint (INP)
  • Time to First Byte (TTFB)
  • First Contentful Paint (FCP)

Learn more about the CrUX Pages Leaderboard report.

Our Test Agents are now using Chrome 123 and Lighthouse 11.7.1 to test your Sites. You can find details surrounding the 11.7.1 release in the official Lighthouse changelog.

Starting in Chrome 122, new functionality introduced a regression that caused Lighthouse to misreport Total Blocking Time (TBT) for pages with long main thread tasks. In these cases, we observed an increase to TBT measurements and long task visualisation.

While the issue has not yet been resolved in Chrome 123+, Calibre took action by disabling the responsible functionality using Chrome feature flags. Additionally, Lighthouse 11.7.1 includes a fix that removes overlapping long tasks from browser trace processing.

Calibre helped to identify the issue and pinpoint the offending functionality that caused the regression in tandem with the Lighthouse core team. With these fixes in place, affected pages will now report accurate Total Blocking Time measurements.

In addition to the above changes, Calibre also adjusted First Contentful Paint (FCP) timeout from 45 seconds to 15 (the Lighthouse default). If you've observed NO_FCP errors as a result of this change, you may want to consider adjusting Test Profile settings so that FCP is lower than 15 seconds. Please contact support for assistence or advice, we're always happy to help.

Our Test Agents are now using Chrome 122 and Lighthouse 11.6.0 to test your Sites. You can find details surrounding the 11.6.0 release in the official Lighthouse changelog.

We know that real user data is a critical part of modern web performance monitoring. In January, we added real user data for Core Web Vitals to Your Sites page. Today, to expand your ability to assess how real visitors experience your sites and apps, we released a Chrome User Experience Report (CrUX) Dashboard:

The CrUX Dashboard is available for all Sites tracked in Calibre. Now you can:

• quickly check the Core Web Vitals assessment status
• track real-user session data without additional costs and scripts
• see if the majority of sessions have a good experience
• observe trends in six key performance metrics (including Interaction to Next Paint, Largest Contentful Paint, Cumulative Layout Shift and Time to First Byte)

Learn more about the CrUX Dashboard report.

React Live Chat Loader now provides full support for HubSpot and Front live chat services.

The latest release (2.9.1) also sees ES5 support reinstated in ESM builds.

Thanks to all the contributors whose efforts helped make this release possible. The full changelog is available on GitHub.

Our Test Agents are now using Chrome 120 and Lighthouse 11.4.0 to test your Sites. You can find details surrounding the 11.4.0 release in the official Lighthouse changelog.

Web performance data can come from a myriad of sources (synthetic testing and real user tracking), both useful for different use cases. Many teams are currently looking for real user data, which is why, we are adding Core Web Vitals data to a refreshed Your Sites dashboard, sourced from Chrome User Experience Report (CrUX).

With the new Your Sites dashboard you can:

• check Core Web Vitals data from synthetic and real-user data sources
• rank websites based on a metric
• monitor a new metric: Interaction to Next Paint (INP)

This change is especially helpful if you haven’t tracked real user data yet, or you’d like to broaden your monitoring with Chrome’s dataset, which is used for Google Search ranking purposes.

Early this year, you can expect a full, chart-based Core Web Vitals dashboard based on Chrome User Experience Report, but for now, Your Sites page is accessible for each Team under Team → Sites (or here).

Learn more about the metrics and methodology on the Your Sites report documentation page.

Our Test Agents are now using Chrome 119 and Lighthouse 11.3.0 to test your Sites. You can find details surrounding the 11.3.0 release in the official Lighthouse changelog.

Lighthouse 11.2.0 removed the “PWA” (Progressive Web App) category for its tests. Inline with this change, Calibre no longer records PWA score metrics. You can access historic PWA scores and audits by downloading pre-11.3.0 reports via the Calibre API, or by downloading lighthouse.json from the Calibre UI.

Single performance tests are a great way to find the reason for observed metric changes that might be caused by requests or asset change. Because we see Snapshots as a critical discovery tool, today we’ve released numerous improvements to Snapshot Overview that make investigating the performance of a page much easier.

With the new Snapshot Overview, you can:

• Easily check the web vitals assessment status
• Check test progression and address authentication errors
• Correlate key paint metrics to render screenshots
• Analyse long tasks and assets for first and third-parties
• Customise the request table display and find requests easier

View the new Snapshot Overview by choosing a test in Site → Snapshots. You can also learn more in the release blog post.

We released a new patch version of our React Live Chat Loader package. In this release, we bumped outdated dependencies and addressed a bug that had caused the distribution bundle to drop support for ES5.

See the full changelog on GitHub.

Over the last few months, we’ve made significant additions to Pull Request Reviews (you can go through all of the changes here).

Today, we have overhauled the performance report posted by Calibre in Pull Requests. Now, the report clearly features:

• Web Vitals assessment status and metrics
• Changes to assets transferred
• Changes to Performance Budgets (if set)

With this information in place, your team can quickly assess the impact of proposed on work:

Find more information in our Pull Request Reviews documentation.

Calibre Test Profiles allow you to specify an emulated device and network throttling for your tests. Until today, all device and network throttling was applied during runtime, meaning that the throttling was applied to the browser as it was running the test.

Google’s PageSpeed Insights uses a different approach. It runs page tests at full, unthrottled speed, then performs a simulation of the throttling that would have been applied to the page load.

Today, we have added a new option to Calibre Test Profiles that allows you to run your tests with the same simulated throttling as PageSpeed Insights. Now, there are two new Test Profile options:

• PageSpeed Desktop
• PageSpeed Mobile

These two device options will run your tests with the same throttling settings as PageSpeed Insights. While we don't recommend comparing monitoring from different tools, this option will allow you to compare your Calibre tests with PageSpeed Insights tests more closely as the test conditions will be more similar.

To start testing with PageSpeed simulated throttling, create or update an existing Test Profile in Site → Settings → Test Profiles.

Our Test Agents are now using Chrome 117 and Lighthouse 11.1.0 to test your Sites. You can find details surrounding the 11.1.0 release in the official Lighthouse changelog.

The 5.1.0 release of Calibre CLI introduces new commands to help you automate your performance testing with more ease than previously possible.

Pull Request Reviews

Designed for use with CI/CD systems, CLI Pull Request Reviews are a way for Calibre to compare production (or another environment) to a preview deployment (e.g.: your work-in-progress branch). Each Pull Request review generates a comparison report outlining any key changes in site speed, which can be used to make decisions about whether to merge the branch or not.

You can automatically fail builds when performance budgets are no longer met using the --failOnUnmetBudget flag, too!

1calibre site create-pull-request-review \
2  --site "my-shopfront-production" \
3  --title "(feat): add lightbox to PDP" \
4  --url "https://pr-123.example.com" \
5  --branch "feat/add-lightbox-to-pdp" \
6  --sha="1234567890abcdef1234567890abcdef12345678" \
7  --waitForResult \
8  --failOnUnmetBudget \
9  --markdown > comparison_report.md
10

Read the Pull Request Review CLI documentation for more information.

Improved formatting for Single Page tests

We’ve overhauled the Single Page Test summary terminal output to include a test summary including Web Vitals assessment and key request sizes:

Read the Single Page Test CLI documentation for more information.

Generate a markdown report for Single Page Tests

Single Page Tests now generate a markdown-powered performance report. You can use this output with other tools that support markdown rendering like GitHub, GitLab and Bitbucket to ensure that all the code you release is reviewed for performance and best practices.

Read the updated documentation for calibre test create and calibre test show commands.

Single Page Test webhooks

When creating a Single Page Test, you can now specify a Webhook URL to post test results to. Using this new technique, you can run async tests and have the results delivered to a custom endpoint of your choosing. You can use --webhookSecret to verify the incoming webhook using HMAC.

1npx calibre test create https://calibreapp.com \
2  --location Sydney \
3  --webhookUrl="https://webhooks.example.com" \
4  --webhookSecret="my verification secret"
5

Read the updated documentation for calibre test create command.

Security updates for dependencies

As part of our regular release cycle, we have applied dependency updates that include important security patches to those projects.

Check full changelog on GitHub.

If you are using Pull Request Reviews to test the performance of your changes in GitHub, you can now be even more proactive with enforcing budget limits.

We added an option to fail Calibre’s budget checks if the pull request exceeds the budget that’s currently passing on the production site. Turn pull request fail checks in Site → Settings → Integrations → GitHub.

Having automations in place, monitoring a non-critical site or a preference for viewing data in-app rather than receiving emails are just a handful of reasons why you might like to create Performance Budgets without email notifications.

All new performance budgets by default have no notifications turned on. If you’d like to receive them, subscribe to relevant Sites’ alerts in your Email Notifications Settings.

Our Test Agents are now using Chrome 116 and Lighthouse 11 to test your Sites. You can find details surrounding the 11.0.0 release in the official Lighthouse changelog.

Chrome 116 introduced updates that affect how Largest Contentful Paint (LCP) is evaluated and calculated. You can read details of these changes on the official Largest Contentful Paint changelog.

Our Test Agents are now using Chrome 115 to test your Sites.

There are dozens of ways to keep up-to-date with new releases from products your team is using. To make it easy, and accessible without leaving Calibre, we added a notification to the main navigation that signifies unread Changelog posts.

We have several major releases coming in the near future, so it’s a good time to be checking in on what’s new!

Our Test Agents are now using Chrome 114 to test your Sites.

Our Test Agents are now using Lighthouse 10.3.0 to test your Sites.

You can find details surrounding the 10.3.0 release in the official Lighthouse 10 changelog.

From July 4, 2023, test artifacts (lighthouse.json, har.json and other test JSON reports that Calibre stores) will be compressed with the Brotli algorithm. This change brings 15-30% smaller downloads compared to previously used compression with Gzip, which also results in faster downloads.

Customers using the Node.js API or Command Line Interface to access test artifacts should update to version 5.0.4, which has been released as a result of this change. You only need to update if you are using the following functionality, or commands:

• Downloading test artifacts for Single Page Tests
• Downloading test artifacts for Snapshot tests
• Using CLI command calibre test download-artifacts
• Using CLI command calibre site download-snapshot-artifacts

If you have a custom API integration, you will need to ensure that artifacts with a response header Content-Encoding with value br are decompressed using Brotli. Tests recorded before this change will remain compressed with Gzip.

From July 6, 2023, Calibre will require all webhook Integrations to use the HTTPS scheme. Existing Webhook Integrations can continue using plain HTTP until that date, but after July 6, we’ll automatically turn off webhook integrations which have not been updated to use HTTPS.

You can review your webhook integration settings by visiting Site → Settings → Integrations.

We published a patch CLI release to download Test Artifacts stored with Brotli compression.

See full changelog on GitHub.

Our Test Agents are now using Lighthouse 10.2.0 to test your Sites.

The upgrade from Lighthouse 9.x to 10.x introduces several changes to the way Lighthouse calculates performance metrics. You can find details on the official Lighthouse 10 changelog.

Key changes:

• Time to Interactive (TTI) is no longer used to calculate Performance Score.
• Cumulative Layout Shift (CLS) is responsible for 25% of the Performance Score (up from 15%).
• Lighthouse now calculates Time to First Byte (TTFB). Calibre is now using the Lighthouse TTFB calculation, instead of its previous measurements.
• Lighthouse calculates Largest Contentful Paint (LCP) phases, which are surfaced in the Largest Contentful Paint Element audit.
• New Back/Forward Cache audit.
• Lighthouse dropped the inclusion of Chrome-Lighthouse in the User-Agent header; however, Calibre will continue to include this header.
• Previously unknown Third Parties are now identified by the Third Party report. You may notice an increase in Total Third Party Code Transferred or Total Third Party Main Thread Execution Time metrics as a result of newly identified third parties.

As a consequence of first two points, your Performance Score might change.

Our Test Agents are now using Chrome 113 to test your Sites.

Chrome 111 and 112 introduced updates that effect how First Contentful Paint (FCP), and Largest Contentful Paint (LCP) are evaluated and calculated. You can read details of these changes on the official Largest Contentful Paint changelog.

Managing service subscriptions can be unnecessarily difficult. Every so often, you might want to pause payments while your team changes focus, re-evaluates tools or needs a breather. That’s why we introduced the ability to temporarily pause your Calibre subscription.

Organisation administrators can pause subscriptions on the Billing → Overview page.

We resolved an issue where the team member count was not appearing in the Team membership section visible on the People → Edit page.

We have added the ability to sort tables within the Lighthouse audits (Performance, Accessibility, SEO, Best Practices and PWA):

Sortable tables allow you to focus on the metrics most relevant to your requirements and quickly identify areas that need improvement, along with those performing well.

Head to a Snapshot and browse the Lighthouse audits to see the new sortable tables in action.

We have made several improvements to reliability of Pull Request Reviews in GitHub:

• Fixed an issue where Calibre was creating Snapshots in Pull Requests when it was set up to ignore branches (for example: !* in the Deployment branches settings).
• Fixed an issue where Pull Requests that were correctly marked as ignored (by Deployment branches settings) would get tested because the Pull Request was opened as a Draft, then moved to Ready for Review.
• Added a missing informative message where a test result was neutral (there was no action from Calibre’s side).

We refreshed the Third Party page, so you can quickly assess the impact of external tools and scripts on performance. The report visualises how first and third-party resources contribute to page transfer size and main thread execution time. You can also quickly pinpoint which category of tools is the most costly, whether it’s bandwidth or time.

Soon, we’ll be adding more information about third-party requests, including details about file types. You can view the Third Party report by navigating to a Snapshot and clicking the Third Party tab.

It’s now easier to prioritise which improvement possibilities to focus on based on Lighthouse audits. You can view the Opportunities and Diagnostics for all Lighthouse audit types (Performance, Accessibility, SEO, Best Practices and Progressive Web Apps) for each test.

For Performance audits, you can filter by Web Vitals metrics impacted by the audit and check the potential time and byte size savings you could observe by addressing specific advice given by Lighthouse.

Check out the refreshed audits by heading any selected Snapshot!

Our Test Agents are now using Chrome 109 to test your Sites.

Our Test Agents are now using Chrome 108 to test your Sites.

We published a new patch version of React Live Chat Loader (2.8.1). In this release, we realigned react-live-chat-loader placeholder styling with Intercom’s current default styling (#215) and bumped all outdated dependencies.

See the full changelog on GitHub.

Our Test Agents are now using Chrome 107 to test your Sites.

Our Test Agents are now using Lighthouse 9.6.8 to test your Sites.

We fixed a problem preventing organisation invitation emails from being delivered in the case where the Resend invitation button was pressed multiple times by the same person to send another invitation email.

Previously Calibre recommended that Time to First Byte (TTFB) should be below 150ms for a good user experience. The recommendation has now been updated to < 300ms based on global data from the Chrome User Experience Report (CrUX).

In Calibre, you will now see a warning if your TTFB is above 300ms and a critical error if it is above 800ms.

Our Test Agents are now using Chrome 106 to test your Sites.

We published a new, minor version of React Live Chat Loader (2.8.0). In this release we:

• Add containerClass prop to allow custom CSS classes on the chat container element
• Improve Typescript definitions
• Refresh the documentation and demo website
• Update placeholder styles to reflect the current default styling of all supported chat beacons

These changes will make using and contributing to React Live Chat Loader easier.

See full changelog on GitHub.

We improved the badge element contrast to pass Web Content Accessibility Guidelines (WCAG) AAA (enhanced contrast) accessibility audits. Previously, they only met the standard on AA (minimum contrast) requirement.

On top of that, we also slightly amended the appearance of badges to give a more soft, rounded feeling. You can see them in action on the Snapshots list, Snapshot page, and Budgets.

We’ll continue evaluating and improving contrast and other accessibility features across Calibre.

Our Test Agents are now using Lighthouse 9.6.7 to test your Sites.

React Live Chat Loader now supports React 18, thanks to contribution from Chris Schuld.

Our Test Agents are now using Chrome 105 to test your Sites.

We published a patch CLI release with a fix to Windows support.

See full changelog on GitHub.

Our Test Agents are now using Chrome 104 and Lighthouse 9.6.4 to test your Sites.

Our Test Agents are now using Lighthouse 9.6.3 to test your Sites.

We published a patch CLI release with the following changes:

• Updated calibre device-list command to output additional emulated device details (screen width and height, device type: phone tablet or desktop and status: current or discontinued)

See full changelog on GitHub.

We continue to improve our open source projects’ governance. React Live Chat Loader, Image Actions and the Calibre Command Line Interface, and Node.js API are now using GitHub Issue Forms. While Issue Forms are currently in beta, they offer a more pleasant and guided way to submitting bugs and questions. See them in action below:

Our Test Agents are now using Chrome 103 to test your Sites.

Our Test Agents are now using Lighthouse 9.6.2 to test your Sites.

Our Test Agents are now using Chrome 101 to test your Sites.

In synthetic and real user monitoring, it’s critical to understand the conditions in which people browse your sites so that we can correlate it to performance. It’s also not uncommon for metric definitions and collection methods to change or bugs to emerge.

To improve the transparency of test settings and debugging changes, we added the Chrome and Lighthouse versions for each test to the Snapshot and Single Page Test report pages:

In 5.0.1, we added a new command, calibre team list, that lists all Teams your API Access Token can access. We also rewrote all command descriptions for better clarity.

See full changelog on GitHub.

You can now see your real user data for Time to First Byte (TTFB) and Interaction to Next Paint (INP) in the Core Web Vitals Checker:

Time to First Byte is critical to track as it can have a cascading, adverse effect on page load and speed. Interaction to Next Paint is a new, experimental metric measuring the longest observed delay between user interaction and the response on your page. While these metrics are not included in the Core Web Vitals set now, we’d recommend adding them to your “to track” list.

Generate your free report here!

We believe transparent governance is critical to any open source project. We updated our most popular projects (React Live Chat Loader, Image Actions and the Calibre Command Line Interface and Node.js API) to reflect the same policies, such as Code of Conduct, security, contribution and issue logging guidelines.

We also updated project READMEs accordingly to make usage and collaboration even more accessible.

Our Test Agents are now using Chrome 100 to test your Sites.

When you add a new Site to monitor in Calibre, we send a summary email with crucial page speed metrics and how you rank against our measurements dataset. To provide a more comprehensive picture of web performance, alongside Web Vitals, we added Time to First Byte to the email report:

We published a new, major version of Calibre’s CLI (5.0.0), now written in ES Modules! With this release we:

• Add a new flag for calibre test create: --waitForTest
• Add new package exports: cli-commands and cli-metadata (to return command functions and documentation)
• Add CLI_COMMANDS.md that lists all available commands with their options for easy reference
• End support for Node 12

From the 5.0.0 version, we’re also ceasing to update the standalone binary builds for the CLI. They will remain accessible for now, but the recommended method of installing an up-to-date version of the CLI is npm.

See full changelog on GitHub.

Setting Cookies and Headers to either authenticate to your Sites or control the test environment has been in Calibre for some time (set via either the Test Agent or Test Profiles). Now, you can specify those settings while using Pull Request Reviews too:

1version: 2
2pullRequestReviews:
3  # Headers
4  - headers:
5      - name: X-Calibre-Test
6        value: Pull Request
7      - name: User-Agent
8        value: Calibre
9
10  # Cookies
11  - cookies:
12      - name: seen-cookie-notice
13        value: true
14      - name: uid
15        value: 1
16        domain: calibreapp.com
17        path: /
18        secure: true
19        httpOnly: true
20

This is especially useful if you’d like to set cookies and headers only for Pull Request Reviews tests (not inherited from Test Profiles settings). For example, you could set a header to signify that the test is for a Pull Request and should be safely routed to the staging infrastructure.

Our Test Agents are now using Lighthouse 9.5 and Chrome 99 to test your Sites.

Our Test Agents are now using Lighthouse 9.3.1 and Chrome 97 to test your Sites.

If you’re using cookies and headers (either in Test Profiles or Site Settings → Test Agent area), we now change when they are applied.

Previously, Calibre set cookies and headers post-authentication. Now, they are applied during authentication so that you can control more fine-grained authentication scenarios, such as hiding CAPTCHA, so Calibre can quickly test your Site or application.

We introduced two new currencies (EUR and AUD) and several new payment methods, so your team can choose a way of paying that’s most convenient and least costly. You can change your payment currency at any time:

Because limited payment methods can be a software adoption blocker and a source of accounting issues, we also added several new methods of purchasing a subscription—not all of which rely on credit cards. We now accept:

• Credit Card (all plans)
• Apple Pay (all plans, only in Safari browser)
• Google Pay (all plans, only in Google Chrome browser)
• Single Euro Payments Area (SEPA) Direct Debit (all plans in EUR currency)
• Bulk Electronic Clearing System (BECS) Direct Debit (all plans in AUD currency)

Thanks to SEPA and BECS, customers in Australia and SEPA countries with relevant plans (AUD or EUR) can pay for their subscription directly from a nominated bank account. You can change your payment method in Billing → Payment Method tab.

With the new payment methods and currencies, we switched to invoices generated by our payment provider—Stripe. Your designated Billing email will still receive invoices after each transaction, and each Administrator can access the entire invoice history in Invoice History and Settings.

We made it possible to display the request table (from the corresponding HAR file) even when your tests fail. We’re now surfacing more information about errors that lead to Lighthouse failures directly in the Snapshot view, so it’s easier to find out the exact reason:

If your tests are failing (continually or intermittently), you can also refer to the Why do my tests fail? guide that walks through common scenarios and steps to fix them.

With two minor releases of Calibre CLI, we added new commands and ways to query your data. Site.get and Site.update now allow obtaining and updating given Sites. We added missing fields to Test Profiles, Deploys, Location, Metric budgets, and Page queries so that you can receive and manipulate all test data.

See full changelog on Github.

We continually build on our best practices to protect your account and data. From now on, all new registrations and, periodically, existing users will have to verify their email accounts unless they are using Google as a method of authentication.

Our Test Agents now use Lighthouse 9.1 We upgrade to stable Lighthouse releases as they become available and after they’re thoroughly tested so that you can leverage the newest version.

Our Test Agents now use Chrome 94 to test your Sites. You can always check the Chrome version in the Lighthouse JSON export or by inspecting the Request Headers for the first request in each Snapshot:

Staying on top of subscription costs and reconciling payments is essential for any business. That’s why we’ve made improvements to make this process easier. Now, you can see the upcoming invoice amount on the Billing → Invoice History and Settings page:

Making changes to Test Packs and User Seats could quickly generate multiple invoices. To avoid dealing with too many PDFs, we batch your changes as they happen and release an invoice after a short time delay. Nobody needs more accounting work!

We created a free tool to check if your site passes the Core Web Vitals assessment. With Core Web Vitals Checker, you can see how many site visitors have a good experience and if your site is likely to lose ranking in Google. All based on Google’s Chrome User Experience Report (CrUX) data:

Check your site, share your reports or learn more about the Core Web Vitals assessment.

Each team has different processes. Sending your monitoring data to your favourite services and creating automations can help with saving time and increasing transparency. We built a Zapier integration, so you can connect to over 4,000 supported apps and create custom workflows without any coding required!

We created a handful of boilerplate workflows (Zaps) to try:

• get a direct message in Slack when a performance budget alert fires
• send page speed data to Google Sheets or Airtable
• run a deploy on a new GitHub release
• create a Trello card to investigate exceeded performance budgets

Learn more about the Zapier integration or enable it on your Calibre account today.

Our Test Agents now use Lighthouse 8.5. We upgrade to stable Lighthouse releases as they become available and after they’re thoroughly tested so that you can leverage the newest version.

We deprecated the site get-pulse-metrics command and replaced it with site metrics (a breaking change). We also added a metric-list command that lists all available metrics, their category, advised ranges and recommendations for tracking:

Read the full changelog here.

If you’d like to (or already are) testing performance in GitHub Pull Requests, there is a new way to configure your test settings. With version 2, you can create configuration options for different branches using positive and negative patterns.

This is useful if you’d like to test differently based on branch names, for example, your preview-public branches are publicly accessible, but preview-* are password-protected:

1version: 2
2pullRequestReviews:
3  - branches:
4      - preview-*
5    authentication:
6      password: mysecretpassword
7      formSelector: form
8      passwordSelector: input[type=password]
9      required: true
10  - branches:
11      - preview-public
12    authentication:
13      required: false
14

Or, you could report different metrics based on branch name too! Convert your configuration.yaml to the new, multi-block format to create branch-specific testing conditions.

You can now use the webhook integration to get alerts about your Budgets. Go to Site Settings → Integrations → Add a Webhook to create one and receive a JSON payload on Budget status changes.

Previously, the only way to enable Deployment Tracking was to leverage the Node.js API, CLI or the HTTP Site API. You can now trigger Snapshots and add deployment markers to your charts with Netlify.

You can enable the Netlify integration under Site → Settings → Integrations → Netlify → Add Webhook.

Using integrations can be a powerful way to connect the services you use. But it can be frustrating when an integration stops working, and we don’t know about it.

Now, Calibre sends you an alert when there’s an issue with any of your connected services (such as Slack, Webhooks, GitHub, Netlify) with steps to fix. No more discovering data or notifications aren’t being sent days later!

Our Test Agents now use Lighthouse 8.3. We upgrade to stable Lighthouse releases as they become available so that you can leverage the newest version.

We’re continuing to improve our GitHub integration by adding more options to skip tests or run them on-demand—whenever it suits your development cycle. Now, you can ask Calibre to test (or re-test) your work by posting a comment with @calibreapp run. Similarly, you can say @calibrepp stop to halt future tests in that PR. Too easy!

There are other situations when you might want to skip testing too. By default, Calibre won’t test draft Pull Requests anymore—only ones ready for review.

You can also use positive and negative patterns to specify which branches should be tested or omitted in Site → Settings → Edit a GitHub Integration. For example, ignore dependabot-generated branches or only test release branches:

React Live Chat Loader now supports Chatwoot and five other live chat providers. If you’d like to decrease the negative speed impact of live chat, install react-live-chat-loader through npm and implement it in your React application.

Previously, we displayed Performance Score, First Contentful Paint and Time to Interactive as default metrics in Calibre. This meant that you’d see those three metrics across Your Sites, Pulse, Pages Leaderboard, Pull Request Reviews and Insights reports. Now, Web Vitals (Largest Contentful Paint, Cumulative Layout Shift and Total Blocking Time) are the default metric set.

Web Vitals have been actively improved for several years and provide a holistic view on different aspects of user experience. Google also uses them as ranking signals for page experience. You can see Web Vitals across the previously mentioned areas. You can still customise metrics displayed in Pull Request Reviews, Pages Leaderboard and Pulse if you’d like to amend them. All measurements are colour-coded, so you know if they’re good (green), need improvement (orange) or poor (red).

In the future, we’ll be introducing more ways of view and metric customisation, as well as keeping track of KPIs.

We know it’s essential always to control your data and be able to delete your account without needlessly contacting support. Now, you can delete your personal account in two clicks if you don’t want to use Calibre anymore.

Our Test Agents now use the latest Lighthouse 8 and Chrome 91 to test your Sites. Lighthouse 8 comes with a few notable changes to metric collection and Performance Score calculation. Here’s a handy explainer of what you might notice in your metrics.

We introduced Personal Access Tokens (PAT), so anyone with a Calibre account can use the CLI and Node.js API to retrieve monitoring information and set up automations:

Before, only organisation administrators could create API access tokens, which created a need to give admin privileges to people who shouldn’t necessarily have access to other admin-only areas such as Billing and People Management. Now, anyone within your organisation can create a Personal Access Token with the level of permissions that mimics their account access level. With the introduction of PATs, everyone gains access to Calibre’s APIs.

Administrators can keep track of and manage all access tokens in the API tab:

Personal Access Tokens will become even more helpful with the introduction of Teams. Not only the newly introduced Viewers will be able to retrieve data, but also Calibre will scope everyone but the admin’s access to the Teams they belong to.

Our Test Agents now use the latest Lighthouse release, 7.4. We upgrade to stable Lighthouse releases as they become available so that you can leverage the newest version.

React Live Chat Loader, our open source project that helps mitigate the negative performance impact of live chat tools now supports Userlike. You can install react-live-chat-loader through npm and implement it in your React application.

Curious how significant the speed impact of using React Live Chat Loader can be? We sped up our Time to Interactive by 30%.

Following the API Tokens refresh in January 2021, we added more capabilities to make managing your access tokens straightforward.

Now, you can sort the tokens by name, creator and expiry date by clicking the respective headings. You also can easily find the token by searching for its name or creator in the search field in the upper right-hand side corner.

We refreshed the way you see and manage people within your Calibre organisation. You can now quickly find a person by searching for their name, email, role or authentication method. You can also sort the information by clicking the table column’s name to, for example, see who is an administrator.

You can still edit, remove and invite new people to collaborate on performance, just more easily.

It’s critical to test speed from various contexts that reflect your audience. performance will differ drastically depending on devices, latency, and location. That’s why we added three new test location regions:

• Stockholm, Sweden
• Cape Town, South Africa
• Bahrain

You can change your tests’ location in the Settings → Test Agent → Test location field or when creating a new Site. Find a complete list of Test Agent locations and their static IP addresses here.

We refreshed Snapshots page, listing all of the tests for a given Site. Now, you can clearly see the test status (Scheduled, Testing, Verifying, Completed or Failed) on per-Snapshot and per Page and Test Profile combination basis.

If you want to know how a Snapshot was started (manually through the interface, through previously configured test scheduler or API integrations) or investigate test status, Snapshots page is the best place to find it

Calibre can log into your Sites and applications as if it was one of your users. Until now, when using Form Authentication, we could specify the form, username and password CSS selectors, but not the submit button selector. Our Test Agent would then crawl your Sites to submit the form and successfully log in.

This process wasn’t bulletproof, especially in tracking competition scenarios where you can’t make improvements to the markup. You can now specify the Submit selector, so there are no failed tests due to markup mismatch. You have full control over selecting which elements should be interacted with when authenticating.

We updated Calibre’s Test Agents from Chrome 88 to Chrome 89. The latest version comes with metric improvements to Cumulative Layout Shift and First Contentful Paint.

We released a minor version of the CLI to address a bug where querying for time-series metrics was not respecting filtering by profile. Read the full changelog here.

We updated Calibre’s Test Agents from Chrome 84 to Chrome 88. Because of numerous fixes and changes to the Largest Contentful Paint metric in versions 86 and 88, you will likely see changes to your Largest Contentful Paint (LCP) measurements.

Our Test Agents now use the latest Lighthouse release, 7.0. As always, we upgrade to stable Lighthouse releases as they become available so that you can leverage the newest version.

This is the first major version release since Lighthouse 6 in May 2019, including numerous significant changes, including new audits. As outlined by the Lighthouse Team, you might see PWA and Accessibility scores changes.

If you are using Pull Request Reviews to track speed before releasing to production, you can now trigger tests only based on a specific list of environments, so the reports in GitHub always relate to what matters to you most.

This is especially helpful when using one deployment provider to release multiple projects or sites. For example, at Calibre, we deploy our marketing website using Vercel. Each deploy publishes a preview of the website and our design system in Storybook. We specify the deployment environment to match the marketing website, so our Pull Request Reviews analyse the website, not Storybook.

We refreshed how you manage and generate API Tokens used to authenticate your account when building automations with our APIs. You can see all tokens for your organisation in the API tab, including their creator and expiration.

When creating a new API Token, you can easily select the areas it should access. We display the new token only once, so it’s essential to save it in a password manager straight away. You can also manage your tokens via the CLI and APIs.

Image Actions, our GitHub Action for automatically compressing images in your Pull Requests, is now using the main branch instead of master.

To avoid breakages in the workflow, update your configuration file to mention the main branch.

You can now also add global Headers and Cookies to manage how Calibre’s Test Agent accesses your Sites. Previously, this option was only available through Test Profiles. Now, you can use global Headers and Cookies for Site-wide settings, and Test Profiles for more targeted testing, such as A/B tests for specific devices.

Our Test Agents now use the latest Lighthouse release, 6.4.0. As always, we upgrade to stable Lighthouse releases as they become available so that you can leverage the newest version.

It’s a frustrating but not uncommon for Lighthouse performance tests to fail. We introduced Compatibility Mode to provide you with critical performance metrics even when Lighthouse returns an error, which otherwise would yield no results.

There are several compatibility modes, which will be triggered depending on encountered types of failures in testing. In all scenarios, you will receive performance metrics, but might not be able to see complimentary audits. Learn more about the difference between Standard Mode, Reduced and Minimum Audits.

A minor version of the CLI has been released with a fix to site create-test-profile, enabling JavaScript execution by default. Read the full changelog here.

We refreshed the Site Settings area so that you can set up and manage your testing faster. You can now search for Pages, which is especially helpful when working with larger Sites. You can explore available integrations in the Integrations tab, to connect Calibre to your favourite services and make the most out of your monitoring data.

We redesigned the primary navigation to make it faster and more straightforward to get around Calibre. Now, you can quickly switch your viewing context between both organisations and Sites, as well as create new ones.

Image Actions, our Open Source GitHub Action that automatically optimises images in your Pull Requests, got a significant upgrade and can now be used with more complex workflows.

You can now see the difference in images pre and post-compression, run the optimisation on a selected schedule, run the action in compressOnly mode that allows for more complex combinations and more. Learn about all changes to Image Actions 2.0 here.

We exposed how long it takes for Calibre to test your Pages with Test Duration visible across the interface in the same way as any other trackable metric:

Test Duration portrays the duration of a singular test from its start to end.

When inspecting metric history charts on Pages Leaderboard and Budgets dashboard, you can now see corresponding deploys if you have enabled Deployment Tracking, are using the Netlify integration or Pull Request Reviews:

This information should make it easier to identify potential spikes visualised in the bar charts. Click on an individual bar to inspect changes further in the Snapshot view.

We improved the structure of Lighthouse audits available in Performance, PWA, Best Practices, Accessibility and SEO tabs for each Snapshot. Now, all audits returned from Lighthouse fall within the following categories:

• Passed
• Opportunities
• Informative
• Not applicable
• Items to manually check

With improved categorisation, you can pinpoint which areas to focus on easier.

We launched Pages Leaderboard so you can identify fast and slow areas of your Sites quickly. In the Pages Leaderboard, you can sort Pages within your Sites based on three selected metrics and device type (desktop or mobile). Calibre will show the last 15 measurements for each metric in your Pages dashboard, colour-coded based on recommended value ranges, so you know which areas need improvements most.

We encourage customising metrics based on your organisational or project goals. For example, you might be wanting to focus on Core Web Vitals. You can also use the Pages Leaderboard for competitive benchmarking, by tracking your landing pages against competitors. With sorting and filtering, you can easily rank Pages against each other.

Learn more about the Pages Leaderboard in the announcement post and our documentation.

Our Test Agents now use the latest Lighthouse release, 6.3.0. As always, we upgrade to stable Lighthouse releases as they become available so that you can leverage the newest version.

There’s a variety of reasons why tests might not be completed successfully—from networking, firewalls, HTTP responses, incorrect markup to timeouts caused by poor performance. Ideally, we spot and address persistent failures quickly, so there’s no disruption to our monitoring and data.

For that exact reason, we introduced Snapshot Failure Emails that alert you when tests are not completing. We will also let you know when tests are running smoothly again.

Anyone can subscribe to Snapshot Failure alerts in Profile → Email Notifications. In case you didn’t see it, we also documented the most common test error types, so you can debug the reasons behind failures more quickly, too.

Our Test Agents now use the latest Lighthouse release, 6.1.1. As always, we upgrade to stable Lighthouse releases as they become available so that you can leverage the newest version.

If you’re testing Sites that require authentication, you can do so without providing the username and rely solely on the password. We know authentication comes in various shapes and forms, thus providing the username to the Calibre Test Agent is now optional.

You can always adjust your authentication settings in Site → Settings → Agent Settings.

Suppose you are using multiple deployment methods for GitHub repositories connected to Pull Request Reviews. In that case, you can now choose the provider that should be used as a basis for performance comparison.

Selecting a deployment provider when many are present will ensure the most accurate results in your performance reports. You can choose a provider in Site → Settings → Integrations → GitHub.

We completely revamped performance budgets to guide you in setting thresholds based on recommended values and your context. With new Budgets, you’re able to see all of your targets with their current status in one place—the Budgets dashboard.

When creating a Budget, Calibre will now showcase the desirable value ranges alongside your past measurements. That way, you can make an informed decision about what’s achievable in the short term, and where would you like to be in the long term. We take out the laborious process of guessing what a good metric value should be.

You can also view your Budgets and see historical data for each Page and Test Profile combination. By doing so, you will be able to easily spot outliers that might need separate thresholds or identify where more performance work is required to meet your goals.

As always, Calibre will notify you about the status of your budgets through email and Slack, based on a selected frequency. In Budgets 2.0, we also identify at-risk values, so you know when a metric is close to exceeding the budget and can address it before it happens.

Read more about what’s new in Budgets and learn how to set them up.

If your deployment previews are password protected and you’re using Pull Request Reviews, you can now add authentication settings to your config.yml to access those environments. Those authentication settings will only be used for Pull Request Reviews tests. You can still specify separate authentication methods for your scheduled and API-invoked Snapshots.

On May 26, we updated our Test Agents to the latest version of Lighthouse. Lighthouse 6 comes with a completely revamped Performance Score algorithm, new metrics, audits, and more. All tests from May 26 onwards run using Lighthouse 6, which is why you are likely to observe changes to your Performance Score.

With Lighthouse 6, we are also reporting a new metric—Cumulative Layout Shift. Alongside Total Blocking Time and Largest Contentful Paint (present in Calibre since September and October 2019), CLS is a part of a new generation of performance metrics that aim to portray user experience more accurately.

This also means that you can track all Web Vitals (Total Blocking Time, Largest Contentful Paint and Cumulative Layout Shift) in Calibre. We are committed to keeping our platform on par with the newest developments in the performance space.

Using the Long Task Timeline is a great way to identify potential issues with JavaScript performance. We updated the visual representation and categorisation of tasks to make it more clear.

The Long Task Timeline showcases all long task warnings and blocking times in the lifespan of the main thread activity. This means that you can not only inspect tasks that contribute to Total Blocking Time (defined between First Contentful Paint and Time to Interactive boundaries) but other long tasks on the main thread that don’t fall in this category.

React Live Chat Loader, our open source component that mitigates the negative performance impact of live chat tools now supports Drift.

If you’re using Drift as your support or sales platform, and React powers your site or app, install React Live Chat Loader to improve your performance scoring.

Previously, when your trial has lapsed, or you have run out of your monthly test allocation, Pull Request Reviews would stop running. Now, Calibre will issue a comment informing you of the necessary steps to be taken to resume testing on your Pull Requests.

This will be especially helpful in contexts where developers rely solely on the GitHub integration versus also logging in to Calibre, where this messaging was always present.

You can now see how many tests were generated with Pull Request Reviews. If you are an administrator, head to Billing → Overview and click the Show per Site usage link.

The test usage will be split between Snapshots and Pull Request Reviews.

A minor version of the CLI has been released with the addition of milliunit measurement support. This will be necessary to report the Cumulative Layout Shift metric once Lighthouse 6 is available.

React Live Chat Loader, our open source component that mitigates the negative performance impact of live chat tools, now supports Facebook Messenger.

If you are using Intercom, Help Scout or Facebook Messenger in your React-powered site or application, you will be able to use the component out of the box and improve your performance metrics. To see its benefits, read about how we improved the speed of our live chat by 30%.

When creating or editing a Test Profile, we now highlight when a Third Party was detected last. That means it’s easier to discover if a Third Party should still be included or not in your blocklists.

We have revamped the Billing area to make it easy to see your test allocation usage, manage Plans, tax information and receipts.

In Billing → Overview you can now see per Site test usage as well as delete your organisation.

You can add or update your default credit card under Billing → Payment method. In Billing → Receipt History and Settings you can ensure your receipts include all necessary information for your tax department and accountants.

At any point, you can adjust your Plan accordingly to your usage.

When using Calibre, you are always in control of your monitoring data. We encourage exporting and building custom integrations with the help of our Node.js API and command-line client.

Exporting directly from the Calibre interface is now even more accessible. You can quickly export metric or Snapshot data from Pulse, Metric and Snapshot views. You can download a CSV, Lighthouse JSON or your page video render (depending on the context), or copy a pre-baked code snippet to use with the CLI.

We redesigned the email you receive after adding a new tracked Site to Calibre. Now, you can see three key metrics—Performance Score, Time to Interactive and Largest Contentful Paint and how they compare to the Calibre test dataset. This change should bring more context into where your Sites are at from the very first minutes of testing.

Additionally, all of our other emails also got a splash of new paint to match.

The CLI has seen two patch releases with the following changes:

• Allowing the use of token store for Node
• Fix parsing of to and from arguments in CLI get-pulse-metrics command

If you want to be up-to-date with releases as they are published, follow the changelog.

We launched Test Packs to allow increasing testing capacity without upgrading your Plan. Each Test Pack adds 5,000 tests to your account and can be added or removed at any time, depending on your needs.

Learn how to use Test Packs or add one in Billing → Overview by clicking the Change plan and manage Test Packs button.

New major release of the CLI brings more functionality, customizability and better error handling. You can now:

• add and manage tokens
• get Agent Settings
• create Sites with --schedule and --interval options
• see relevant errors when trying to obtain data from a non-existing Snapshot

Read the full changelog or browse the API documentation.

Max Potential First Input Delay is calculated based on First Input Delay (FID). It portrays the maximum time it takes the browser to respond to a user action, such as clicking a button or trying to type something into a form. Max Potential FID can be used alongside Time to Interactive to quantify when your sites and apps can be successfully interacted with.

You can add Max Potential First Input Delay to the Pulse page through customising the Metric History view and set budgets against it.

Most performance and user experience issues are discovered when it’s already too late—long after they have already impacted your customers. Pull Request Reviews allows you to identify speed bottlenecks before they reach production, directly in your Pull Requests.

If you’re using GitHub and one of the following deployment methods: Vercel, Heroku, Netlify or GitHub deployment statuses, you will be able to enable Pull Request Reviews and receive detailed performance reports on your work-in-progress. Read the announcement or learn how to get started.

When Lighthouse reports potential savings, you can now inspect their details to find out where exactly to look and how much time or bandwidth is there to be saved.