AI Workflow Automation: Where AI Actually Fits in a Business Process

The symptom: the pipeline looks fine and the site is not

The failure mode of AI workflow automation is rarely a crash. A crash is easy. The failure mode is a run that exits zero, writes a row to the log, and leaves the site in a worse state than before, with nobody looking.

In the content pipeline we maintain for a client, the symptom showed up as a line in a daily log that repeated itself. The post-publish checklist printed slug not in sitemap for one article, then two, then four. Each run finished without an exception. The cron job at 12:00 Monday to Friday kept publishing. The pages returned 200 to anyone who visited them. Nothing paged anyone, because nothing had failed in the sense a monitoring tool understands.

That is the shape of the problem. Someone notices it because a human reads a log line that has appeared too many times in a row, or because a search console report shows a page that exists but has no discoverable path to it. The pipeline itself has no opinion.

How to diagnose it

Start by separating the steps that produce text from the steps that change state. In this pipeline the sequence is fixed:

• Spreadsheet row: topic, channel, pillar, priority, evidence notes
• Prompt assembled and sent to a language model
• Model returns text
• Text converted to HTML
• Validators run
• Publish
• Sitemap update
• Hub page rebuild
• IndexNow ping
• Post-publish checklist

Only step three involves the model. The other eight are ordinary Python. So the first diagnostic question is not "what did the model do wrong", it is "which deterministic step assumed something that stopped being true".

Concrete checks that found the sitemap bug:

• Fetch the sitemap and inspect the root element. curl -s https://example.com/sitemap.xml | head -c 400. If the root is <sitemapindex> instead of <urlset>, every append written by the pipeline is going into the wrong document.
• Check whether the article slug appears in the sitemap: curl -s https://example.com/sitemap.xml | grep -c "the-slug". Zero means the page is orphaned from discovery.
• Read the log for repeated lines rather than errors. grep -c "not in sitemap" pipeline.log. A count that grows every weekday is a signal, even when no line says "error".
• Confirm the page itself is healthy: curl -s -o /dev/null -w "%{http_code}" https://example.com/the-slug. A 200 with no sitemap entry is the exact combination that hides the problem.

For the language issues, the diagnostic is a diff. Take a generated article and run it through the dictionary check by hand, then count the corrections. If the correction count per article is non-zero and varies, the model is not reliable for that job and should not be trusted with it.

What the data showed

The sitemap failure had a specific mechanism. The sitemap file became a sitemap index on 18/09. The publishing engine still expected a <urlset> document and appended new <url> entries by locating the closing </urlset> tag and replacing it with the new entry plus the tag. Once the file was a sitemap index, that closing tag no longer existed. str.replace on a missing substring does not raise. It returns the original string unchanged. So four articles were published with no sitemap entry at all, and the checklist warned slug not in sitemap every day until someone read the log.

That is the whole failure: a silent no-op in a string operation, combined with a checklist that was correct but unread.

The second failure was in the channel filter. The code selected rows where the channel string contained artigo. The first English topic had channel artigo_en, which contains artigo as a substring, so the filter would have routed an English topic into the Portuguese publishing path. It was caught before publication, and fixed by switching from substring matching to exact comparison.

On the language model itself, the observed behaviour is that it drops or misplaces Portuguese accents unpredictably. Not always, not in a pattern that a prompt can address. That is why a 92,000-word hunspell dictionary corrects the text after generation. The model is not trusted for orthography. It is also not trusted for numbers: the prompt forbids any number not present in the spreadsheet's evidence column, and em dashes and emojis are stripped to match house style.

Where each step belongs

The fix

Three changes, in order of importance.

One: make state changes fail loudly. Replace string surgery on XML with a parser. If the root element is not what the code expects, raise. A pipeline that stops is better than a pipeline that publishes into a void.

import xml.etree.ElementTree as ET

tree = ET.parse("sitemap.xml")
root = tree.getroot()
if not root.tag.endswith("urlset"):
    raise RuntimeError(
        f"sitemap root is {root.tag}, expected urlset. "
        "Refusing to append."
    )

The same principle applies to the channel filter. Exact comparison, not containment:

if row["channel"] == "artigo":
    route_to_portuguese(row)
elif row["channel"] == "artigo_en":
    route_to_english(row)
else:
    raise ValueError(f"unknown channel: {row['channel']}")

The else branch matters more than the two comparisons. An unknown channel should stop the run, not fall through to a default.

Two: keep the model on a short leash. The model receives a prompt built from the spreadsheet row and returns text. It does not choose the channel, the slug, the publish time, or the sitemap. Those are inputs. The validators run after generation and before publish, in this order: dictionary correction for Portuguese accents, house style stripping, number check against the evidence column.

Three: make the checklist impossible to ignore. The checklist already did its job. It printed slug not in sitemap for four consecutive days. What was missing was a threshold. If the same warning appears on two consecutive runs, the pipeline should stop publishing and surface the line somewhere a person will see it, not append it to a log file.

How to verify the fix worked

Before the fix, the measurable facts were: four articles published with no sitemap entry, and a warning line appearing once per weekday for four days. After the fix, the same measurements should read zero and zero.

What to check, in order:

• Run the pipeline against a sitemap index on purpose. It should raise, not publish. This is a one-line test and it is the test that was missing.
• Count slugs present in the sitemap versus pages published in the last seven days. The two numbers should match. grep -c "<loc>" sitemap.xml against the count of published rows.
• Count dictionary corrections per article. A stable number means the model's behaviour is being caught by the same net every time. A number that jumps means the prompt or the model changed.
• Count occurrences of em dashes and emojis in published output. Should be zero, and should stay zero without anyone checking.
• Count numbers in published articles that do not appear in the evidence column. This one needs a human, because the check is semantic. Keep the sample small and the review short.

The general verification rule: for every step the model touches, there is a deterministic check downstream that can fail. If a step has no such check, the model should not be in that step.

Frequently asked questions

If you are mapping where a model belongs in a process you already run, we wrote up the approach we use at prospectia.space/en/automation.