Background jobs keep calling a deactivated site whose identity endpoint still answers
Description
Problem
AP-2379 recognises a removed site by its identity endpoint (/_edge/tenant_info) answering "not found". A deactivated site still answers there with its cloud id, while every Jira and Confluence request answers 404 with the HTML page "Atlassian Cloud Notifications - Page Unavailable". Such a site is not treated as unavailable: its installation stays installed and enabled, the background jobs keep processing it, and every failure is logged as an ERROR or WARN.
Example on QA: host 29, pn-test-1.atlassian.net, unused for a long time. Checked 2026-11-18: /_edge/tenant_info answers 200 with the cloud id; /status, /rest/api/3/serverInfo and /wiki/... answer 404 "Page Unavailable". Log:
UserCacheService: Failed to fetch Jira users for host 29: GET https://pn-test-1.atlassian.net/rest/api/3/user/bulk?... failed with HTTP status code 404
The site has no Forge system token (an old Connect installation), so the uninstall after 7 days without a token refresh never applies to it. The same can happen on production for any customer site that gets deactivated while the app is still installed.
A pending approver refresh that failed this way was picked up again by the next stale sweep about 35 minutes later, forever, because claiming an approval only leases it for 30 minutes.
Change
-
Before the app calls Jira or Confluence for a site, it checks whether the site is deactivated: one anonymous
GET <site>/status(<site>/wiki/statusfor Confluence), without following redirects. Only a 404 with the "Page Unavailable" page counts as deactivated; any other answer, or no answer, counts as active. An active site is checked again after 12 hours, a deactivated one after 30 minutes. Only the first request for a site waits for the first check, at most 3 seconds. -
A deactivated site is handled exactly like a removed site (AP-2379): background jobs skip it with one INFO line a day per job and site, no ERROR or WARN; notifications are marked failed without retry; email decision links, Slack buttons and the Web API answer "This site is no longer available."
-
The log lines for unavailable sites now say "(removed, renamed or deactivated)" instead of "(removed or renamed)".
-
A pending approver refresh that fails permanently (Jira or Confluence refused it with a 4xx other than 408 and 429, or the site is gone) is tried again only after the regular 12 hours instead of about every 35 minutes. Other failures (5xx, 408, 429, connection problems) are still retried after about 30 minutes.
-
A 408 answer counts as temporary, like 429 and 5xx.
-
One rule for "Atlassian refused the request" (4xx except 408 and 429) is shared by the pending approver refresh and the work item field push.
Fixes included
-
An active approval whose work item or page the app can no longer reach (Jira or Confluence answers 404: deleted without the delete event reaching the app, archived, or hidden from the app by permissions) flooded the log on every expiry and reminder run: the expiry job, the notification preparation and the Jira notify call each logged an ERROR with a full stack trace. Example on QA, host 22: "Failed to fetch work item: 1342225 - Issue does not exist or you do not have permission to see it." and "Failed to process VALIDITY expiration for approval 428891". Each of these failures is now one WARN line with the approval, work item or page and the Jira or Confluence message, without a stack trace. The approval itself is not changed (a 404 does not prove the work item is deleted), and the notification error is still recorded for the site's admin.
Decisions
-
A deactivated site is not disabled in the database (same as AP-2379): when a site is reactivated, the app works again for it within about 30 minutes.
-
The check only reacts to the "Page Unavailable" page, so a live site that answers 404 for another reason (a JSON error or Jira's own "dead link" page) is never treated as deactivated.
Acceptance criteria
-
A deactivated site produces at most one INFO line a day per job and site, and no ERROR or WARN lines.
-
Live sites behave as before. An approval whose work item or page answers 404 logs one WARN line without a stack trace per expiry, reminder or notification, instead of ERROR stack traces.
-
A pending approver refresh refused with a 4xx is not repeated before 12 hours.
Manual tester notes
A deactivated site cannot be prepared on QA by a tester, so these are regression checks only.
-
Open Approval Path on a Jira work item and on a Confluence page: both load normally.
-
Start an approval and approve it from the panel: the decision is recorded.
-
Approve from the link in an approval email: the decision is recorded and the confirmation page shows.
Before release
-
After the QA deploy, check the QA logs: no more ERROR or WARN lines for host 29 (pn-test-1.atlassian.net); instead at most one INFO line a day per job, for example "Site does not answer at its stored URL (removed, renamed or deactivated): Pending approver reconciliation skipped Jira site https://pn-test-1.atlassian.net (host 29)".
-
After the QA deploy, check the QA logs for host 22: no more ERROR lines "Issue does not exist or you do not have permission to see it." from NotificationExceptionHandler, JiraNotificationErrorLog or ExpirationScheduler; the same failures appear as single WARN lines.
-
The AP-2379 release check looks for the old text "(removed or renamed)"; after this change the lines say "(removed, renamed or deactivated)".