Inside AIRO: Building an Extensible AI Content Audit Framework for Drupal

In our previous article, we explored why AI-assisted discovery requires organizations to think beyond traditional search optimization.

The next question is more technical:

How do you evaluate AI readiness inside a complex Drupal platform without separating the analysis from the content workflow?

That was the engineering challenge behind AIRO. We wanted to create more than a prompt attached to a content form. The system needed to understand rendered Drupal content, work with different AI providers, produce structured and actionable assessments, support individual pages and entire sites, respect Drupal’s access model, and remain extensible as AI discovery practices evolve.

This is how we approached it.
 

Starting With a Modular Architecture

AIRO is organized into three layers with different responsibilities.The main AIRO Preview module provides the editor-facing experience. It adds an analysis panel to supported Drupal content and allows an authorized user to ask questions about the current page or, when configured, content from across the website.The AIRO Scoring module adds structured assessments, readiness scores, checkpoints, prioritized action items, technical audits, saved results, and assessment history.

The AI Site Audit module aggregates those individual assessments into a sitewide view. It identifies patterns across content types, displays score distributions, surfaces common failures, and provides strategic insights based on the collected results. Separating these concerns was intentional. A team may want the preview experience without running automated assessments. Another organization may need scoring and technical checks but choose to delay sitewide analysis. The modular architecture allows each Drupal project to adopt the capabilities it needs without forcing the entire system into a single installation path.
 

Building on the Drupal AI Ecosystem

We did not want AIRO to be tied to one model vendor. The module builds on Drupal AI’s provider architecture, allowing a site to use a compatible chat provider and model already configured within Drupal. Administrators can establish defaults, while trusted users can be allowed to compare different provider and model combinations.This approach provides two advantages.

• First, it reduces duplicated integration work. Provider credentials, model discovery, and chat operations remain part of the Drupal AI ecosystem.
• Second, it gives organizations control. A team can select a provider based on security, privacy, cost, performance, hosting, or governance requirements rather than being locked into a decision made by the module.

AIRO focuses on the content analysis workflow. Drupal AI manages the connection to the model.
 

Extracting the Content the User Actually Sees

One of the first engineering challenges was deciding what content should be sent for analysis. Drupal content does not always live in a single body field. A page may contain standard fields, entity references, media, taxonomy, paragraphs, computed values, or sections assembled through Layout Builder. Analyzing raw field values alone can miss the structure and relationships that give the page meaning. Analyzing only raw HTML can introduce markup noise and obscure the content signals we want the model to evaluate.

AIRO uses an extensible content extraction layer to separate the analysis workflow from the way content is stored. The extraction process can represent headings, links, images, metadata, and entity context using consistent markers. This gives the model a clearer view of the rendered page structure while preserving information that matters for the assessment.The module is also revision-aware. If an editor is reviewing a specific revision, the analysis must use that revision rather than silently loading the currently published version.

That detail is essential in an editorial workflow. A useful preview must evaluate the content the editor is actually working on.
 

Designing the Current-Context Preview

The simplest AIRO workflow begins with the current page.An editor opens the AIRO panel and asks a question. The module extracts the content from the active node revision, combines it with configurable system and user prompts, and sends the request through the selected Drupal AI provider.

The default prompt asks the model to respond as a visitor who has read the page. It also instructs the model to use only the provided context and identify information that is unclear or missing.This constraint is important.Without it, the model may answer from its general training data, creating the impression that the page communicates information that does not actually appear in the content.

AIRO is not intended to test everything a model knows about an organization. It is intended to test what the organization’s content successfully communicates.
 

Expanding the Context With RAG

A single page does not always contain the complete answer. Organizations frequently distribute information across service pages, policies, articles, documentation, biographies, and related resources. To support those cases, AIRO includes a Whole Site context option built around retrieval-augmented generation.

The workflow uses Search API and Drupal AI Search to retrieve relevant content from a configured index. An embeddings provider converts content into vector representations, while a compatible vector database stores and retrieves the relevant chunks.

When a user asks a question, AIRO retrieves the most relevant site content and supplies it to the same preview workflow as contextual evidence.

The module does not require a specific vector database provider. Teams can choose the option that fits their infrastructure, including supported services based on Pinecone, Milvus, PostgreSQL, SQLite, or Elasticsearch.

We also kept the two context modes explicit in the interface.

Current Context answers from the page being reviewed.

Whole Site answers from relevant content retrieved across the configured index.

That distinction helps editors understand what evidence the response is using and prevents a sitewide answer from being mistaken for a page-level assessment.
 

Combining Deterministic Signals With Model Analysis

Language models are useful for evaluating tone, clarity, completeness, and semantic structure. They are not always the best tool for counting or detecting technical signals.

AIRO therefore combines deterministic analysis with model-based evaluation.

Before requesting a score, the module calculates signals that can be established directly, such as heading counts and the presence of specific metadata. Those signals are included with the extracted content and assessment instructions.

The model then evaluates dimensions that require greater interpretation, such as:

• Whether the introduction answers the likely user need
• Whether individual sections are self-contained
• Whether the tone remains consistent
• Whether important topics appear to be missing
• Whether links use meaningful language
• Whether image descriptions provide useful context
• Whether the content includes clear definitions or question-and-answer patterns
• Whether sections are suitable for retrieval and reuse

This hybrid approach reduces the amount of basic detection delegated to the model while preserving its ability to analyze meaning and context.
 

Making Model Output Predictable

Free-form AI responses are difficult to store, compare, and display consistently.

For scoring, AIRO asks the provider to return a structured JSON object that follows a defined schema. The response includes an overall readiness score, qualitative status, sub-scores, checkpoints, action items, readability findings, metadata signals, content patterns, and retrieval quality indicators.

The overall score is divided across three dimensions:

• Technical SEO, up to 40 points
• Content quality, up to 35 points
• Schema markup, up to 25 points

The sub-scores must add up to the overall score.

Once the provider returns the response, the module validates and parses the JSON before creating a saved assessment. It also supplies defaults for optional structures when providers omit newer fields, helping preserve compatibility as the assessment format evolves.

Structured output gives the interface predictable data. It also allows sitewide aggregation without sending every full page back to the model.
 

Turning Findings Into a Workflow

A score alone does not tell an editor what to do.

AIRO converts the assessment into checkpoints and action items. Each checkpoint includes a category, status, and priority. Each action item explains the issue, why it matters, the likely target, and, when appropriate, suggested content.

Action items can be marked as completed. The module records completion state while preserving the original assessment, allowing teams to use the analysis as an operational checklist.

Assessments can also be saved historically. When history is enabled, AIRO calculates the score difference between assessments so teams can see whether their changes improved the result.

This shifts the experience from passive reporting to iterative improvement:

1. Analyze the content.
2. Review the evidence.
3. Complete prioritized actions.
4. Run the assessment again.
5. Measure the change.

Creating an Extensible Technical Audit System

AI readiness is not limited to page copy.

A technically inaccessible page, incomplete sitemap, restrictive robots.txt file, missing canonical URL, or absent structured data can undermine otherwise strong content.

We designed AIRO’s technical audit layer as a Drupal plugin system. Each audit check is independently discoverable and executable. New checks can be added without expanding one centralized service or rewriting the assessment workflow.

The current checks examine signals such as:

• HTTPS availability
• robots.txt
• Sitemap configuration
• llms.txt
• Canonical URLs
• Schema markup
• Language declarations
• Date metadata
• Content licensing
• Feed availability
• Entity relationships
• JSON availability

The scoring submodule also includes filesystem-level checks for relevant configuration, security, and site health conditions.

Separating deterministic technical audits from AI-generated content assessments makes the result easier to explain. A failed HTTPS check should come from an observable condition, not a model’s interpretation.
 

Scaling From Nodes to the Entire Site

Running an assessment interactively is useful during editing, but large platforms require automation.

AIRO supports queued assessments and command-line workflows so teams can process content in batches without requiring an editor to open every node. Configuration options help control automatic assessment behavior, history retention, and provider usage.

The sitewide audit layer then aggregates the saved assessments.

Instead of asking the model to repeatedly reread every page, AIRO calculates statistics first. It summarizes assessment coverage, average scores, score distribution, content type performance, sub-score averages, failing checkpoints, and frequently recommended actions.

An optional AI interpretation step receives those aggregated statistics and produces a strategic summary, priorities, quick wins, and projected areas of improvement.

This architecture reduces unnecessary model calls and creates a clearer boundary between measurable data and AI-generated interpretation.
 

Respecting Drupal’s Access and Security Model

Adding AI functionality should not bypass the controls already protecting Drupal content.

AIRO applies node and revision access checks when loading content for the panel. Permissions determine who can use the preview, run assessments, manage prompts, select providers, or access Whole Site context.

Interactive requests use CSRF protection, and the interface prevents overlapping preview queries from producing conflicting states.

These controls are not decorative additions. They are part of the architecture.

An AI feature that can retrieve unpublished or restricted content without respecting Drupal access rules would introduce a serious governance problem. AIRO is designed so that the analysis experience remains subject to the same content boundaries as the rest of the platform.
 

What We Learned

Building AIRO reinforced several principles that apply to AI development beyond this module.

First, AI output becomes more useful when the surrounding system provides structure. Clear schemas, deterministic signals, constrained context, and explicit permissions matter as much as the prompt.

Second, content readiness cannot be reduced to one model response. It requires a combination of editorial analysis, technical inspection, structured data, accessibility, and organizational governance.

Third, AI should support human decision-making rather than hide it. Editors need to understand why a recommendation exists, decide whether it is appropriate, and remain responsible for the final content.

Finally, extensibility matters. AI discovery practices, models, standards, and organizational requirements will continue to change. A sustainable Drupal implementation needs replaceable providers, configurable prompts, pluggable audit checks, and optional functional layers.
 

Looking Ahead

AIRO remains under active development, and the architecture is designed to evolve with the Drupal AI ecosystem and the needs of the open-source community.

Future improvements can expand the audit check library, strengthen deterministic scoring, support additional content structures, improve model observability, and provide deeper sitewide comparisons.

The goal is not to create a system that declares content ready and ends the conversation.

The goal is to give Drupal teams a transparent, extensible framework for asking better questions about their content—and turning the answers into measurable improvements.

AI-assisted discovery is still evolving. The tools used to prepare for it should be able to evolve as well.

About Mobomo, LLC 

Mobomo, a private company headquartered in the D.C. metro area, is a CMMI Dev Level 3, ISO 27001:2022, ISO 9001:2015, and CMMC Level 1 provider of digital transformation system integration services. A premier provider of mobile, web, infrastructure, and cloud applications to federal agencies and large enterprises, Mobomo combines leading-edge technology with human-centered design and strategy to craft next generation digital experience. From private sector companies to government agencies, we have amassed deep expertise helping our clients enhance and expand their existing web and mobile suite. Interested in learning more about Mobomo? Take a tour of our capabilities, our portfolio of work, the team members who make our clients look so fantastic, and feel free to reach out with any questions you might have.