News

The leaders of intelligence agencies from the Five Eyes nations – Australia, Canada, New Zealand, the USA and the UK – have together issued strongly worded advice calling for leaders to nail cybersecurity basics or fall victim to ruinous AI-powered attacks. “The rapid pace of frontier AI development means cyber risk assumptions can become outdated in months, not years,” the advice warns, and calls for organizations to take rapid action to ensure their defenses remain potent. “While AI will help us improve cyber defence over time, it also accelerates the speed, scale, and sophistication of cyber threats,” the advice adds. “Frontier AI models are anticipated to exceed current industry expectations, fundamentally transforming both offensive and defensive cyber capabilities. The timeline is not years, it is months.” After all that scary stuff, the spook bosses offer some antidote: “Cyber resilience is integral to advancing business continuity, market confidence, and long-term value.” And how might one achieve that resilience? The Five Eyes have four suggestions: Understand and assess risk, readiness and accountability Prioritize foundational cyber security practices and controls Empower cyber leaders with authority and resources Stay actively engaged as threats and guidance evolve “Cyber risk can no longer be treated as a purely technical issue,” the advice points out. “This is a core business risk and leadership responsibility,” because breaches are inevitable and “Breaches will occur. Preparedness helps you contain them quickly and prevent escalation into major operational and financial crises.” The intelligence chiefs therefore want organizations to test their cyber resilience rigs. “It is not enough to have controls,” they write. “Leaders must be confident those controls will perform during a real incident. This requires reassessing long-standing trade-offs and using AI deliberately to strengthen defence – not just improve efficiency.” That last sentence is a rare moment of optimism in the advice and precedes a section in which the intelligence bosses observe “Organizations that integrate AI tools into their security operations can detect vulnerabilities earlier, improve software quality, monitor unusual behaviour, and respond faster to incidents – reducing both the cost and impact of incidents.” Readers of The Register might find this advice a little quaint given that infosec vendors have for years blathered on about the need for boards and bosses to take cyber seriously. It’s also been a couple of years since it became apparent that generative and agentic AI can fuel new and unusually potent cyber-attacks. Interest in that idea spiked in the eleven weeks since Anthropic revealed the existence of its powerful flaw-finding Mythos model and hid it behind a regwall lest criminals use it to swiftly slice holes in important software. The Five Eyes bosses address their advice to “leaders” – presumably bosses of substantial organizations – who may not have watched the Mythos mess unfurl while they worried about a global energy crisis kicking holes in their supply chains. The good news is that the spy bosses don’t think leaders need to learn a lot to cope with the advent of AI, as their advice suggests five practical actions they rate as “not new,” but “now urgent to reduce not only technical risk, but also operational, financial and reputational exposure.” For the record, those actions are: 1. Reduce your attack surface: Limit unnecessary system access and external connectivity. Challenge whether systems need to be exposed at all and isolate those that do not. 2. Accelerate patching processes: AI is shortening the time between vulnerability discovery and exploitation. Delays in patching increase risk, especially for operational systems with long update cycles. Prioritize security updates accordingly to manage risks. 3. Address legacy systems: Unsupported systems are easy targets. They are not just technical debt, they are strategic liabilities. 4. Review and strengthen identity and access controls: Limit who can access critical systems. Enforce strong authentication and regularly review permissions. 5. Prepare for incidents before they happen: Test response plans, train and prepare teams, and assume breaches will occur. Focus on fast containment and recovery. Take us, and this, to your leaders, dear readers. ®

The JavaScript development ecosystem may be a security nightmare, but it's also ripe for improvement. One such tool is the CVE Lite CLI, a free open source dependency scanner that helps reduce the risk of software supply chain attacks. It runs locally and provides actionable vulnerability fixes, if any are available. The tool, endorsed by OWASP, has recently been updated to include override auditing, which has the potential to avert transitive dependency vulnerabilities such as the March 2022 node-ipc package incident. The Shai-hulud software supply chain attacks that have been vexing security professionals for the past few months underscore how common it has become for threat actors to target the developer ecosystem, including CI/CD, package registries, and developer tooling. Software developers can reduce their risk by making sure the dependencies in their apps are up to date and free of known vulnerabilities, but that's more difficult than it should be. It's generally apparent when a particular library or module relies on a vulnerable dependency. But there isn't necessarily an available fix or clear remediation path. Modern JavaScript applications, like many other programming languages, allow developers to incorporate pre-existing solutions to particular problems in the form of packages – modular code that can be imported to implement particular functionality. These packages commonly depend on other packages, which is why they're known as dependencies. And these dependencies in turn may also depend on still more packages, referred to as transitive or indirect dependencies. A common security scenario goes something like this: A developer creates an app using some application framework. The app includes a dependency on "Package A", which itself relies on "Package B" – the transitive or indirect dependency in this situation. If the maintainers of "Package B" have deployed a patch addressing a reported CVE, but the maintainers of "Package A" haven't gotten around to incorporating that change into their code, apps incorporating "Package A" may be vulnerable to attack. Among other possible responses, affected developers may choose to create an override to replace the outdated, vulnerable version of "Package B," a configuration entry that can be removed once "Package A" gets repaired. But Sonu Kapoor, creator of CVE Lite CLI, explained to The Register that overrides represent a legitimate security tool but have limitations. "When a transitive dependency has a CVE and the upstream maintainer hasn't shipped a fix yet, you pin it via npm overrides, pnpm overrides, or Yarn resolutions," Kapoor explained in an email. "Once the vulnerability is addressed and CI passes, you move on. The problem is what happens after that." Kapoor recently added an override auditing tool to the CLI. When he scanned four popular JavaScript open source projects, he found that three of the four had broken overrides. "Cal.com has 90 override entries and 11 that are silently doing nothing," he said. "Jest has an override for its own package name pointing at nothing in the resolved tree. NoCoDB has entries using wildcard patterns that never matched any path in the graph. Next.js was the only clean one with zero findings, which tells me the tool is finding a real pattern, not noise." This can be dangerous, he said, when a project migrates between package managers (e.g. npm to pnpm) that looks for overrides in a different location. "npm reads from overrides, pnpm from pnpm.overrides, Yarn from resolutions," he explained. "When a team migrates package managers and forgets to move their security pins, the package manager silently ignores them. No error, no warning, the vulnerable package ships unconstrained." Kapoor said that AI coding assistants commonly advise developers to add override entries when asked to fix a transitive dependency vulnerability. "That advice is correct at the moment," he said. "None of them ever tell the developer to come back and verify the entry still works." CVE Lite CLI, Kapoor said, does not recommend overrides as the way to properly address a vulnerable dependency. "Overrides look like a security fix in package.json, but routinely outlive their purpose – they can point at packages no longer in the dependency tree, apply to the wrong package manager entirely, or shift to an unintended version on every install," he said. "The override hygiene feature exists precisely because of this failure mode: teams add an override to address a CVE, move on, and years later, the override does nothing while they still believe they're protected." ®

OpenAI announced a flurry of cybersecurity-related AI news on Monday, releasing an improved version of GPT‑5.5‑Cyber, its most advanced vulnerability-finding model, along with an expanded partner program for cybersecurity vendors, an update to its Codex Security scanner⁠, and an initiative to “Patch the Planet” – or at least 30 high-profile open source projects. The announcements come as Anthropic’s Mythos mess keeps getting more complicated, with national security concerns clouding defenders’ abilities to use that AI company’s most advanced models to find and fix vulnerabilities – or perhaps it’s just politics as usual. They also coincide with a general feeling of FUD around AI cyberattacks and the impending vulnpocalype. The Reg’s vultures will keep out collective eyes on all of this. First off: GPT‑5.5‑Cyber. After releasing a preview version of the model to a select group of “trusted defenders,” OpenAI on Monday released an update that it says makes the model even better at finding – and also fixing – bugs in code. “It is our strongest model yet for finding and helping patch software vulnerabilities, while retaining GPT‑5.5’s general-purpose intelligence and ability to work across long, complex tasks,” the AI shop said. “The model can sustain deeper analysis across large codebases: identifying security-relevant components, tracing whether vulnerable code is reachable, validating likely issues in controlled environments, developing and testing patches, and preparing evidence for human review.” OpenAI said it evaluated the update and 5.5 preview using a few different benchmarks: CyberGym, which test how well AI systems can reproduce known vulnerabilities; ExploitGym, which determines how well models can turn known vulnerabilities into working exploits that achieve unauthorized code execution; and SEC-bench Pro, which measures AI systems’ long-horizon vulnerability discovery and proof-of-concept generation capabilities. The updated version 5.5 outperformed the preview model in all three tests, we’re told. On CyberGym, the updated GPT‑5.5‑Cyber reached 85.6 percent success, compared with 81.8 percent for GPT‑5.5. On ExploitGym, it outperformed the earlier model 39.5 percent versus 25.95 percent. And on SEC-bench Pro, GPT‑5.5‑Cyber hit 69.8 percent, compared with 63.1 percent for GPT‑5.5. Plus, OpenAI assures everyone that it’s had “ongoing dialogue” with the US government, including about its latest model plus upcoming releases, so hopefully that insulates the company against any surprise export controls. OpenAI also expanded its partner program. The OpenAI Daybreak Cyber Partner Program currently has about 30 security-vendor and service-provider partners, and only these select firms get to use the updated GPT‑5.5‑Cyber model. OpenAI says it plans to add more organizations to the elite group “in the coming months.” FOSS flaw-finding Also on Monday, OpenAI announced Patch the Planet, an initiative to help open source project maintainers find and patch vulnerabilities. This initiative, co-founded with Trail of Bits and launched in collaboration with HackerOne and AI-powered bug hunting outfit Calif, provides participating open source projects with ChatGPT Pro, conditional access to its Codex Security scanner, and API credits for core development, maintainer automation, and release workflows. “Maintainers define their priorities, preferences, and established disclosure processes,” according to OpenAI. “Patch the Planet security researchers then manage the work end to end - validating and deduplicating both vulnerabilities and patches before they reach maintainers, significantly reducing the burden on maintainers and speeding up remediation.” Trail of Bits reports that in the first week alone, Patch the Planet uncovered hundreds of bugs, and generated 64 pull requests with 51 issues filed across 19 projects. The 19 projects Patch the Planet assists includes cURL, NATS, pyca, Sigstore, aiohttp, the Go project, freenginx, Python and python.org, urllib3, PyPI, SimpleX, Valkey, and RustCrypto. More than 30 projects have joined so far, and project maintainers can apply to join the initiative. Some of the initiative’s highlights from the week include using GPT-5.5-Cyber to build a full-scale fuzzing lab in under a day – an effort we’re told would have take human fuzzing experts two or three weeks to do manually. Patch the Planet also used Codex to build a CVE variant analysis pipeline. This also took less than a day to complete. Speaking of Codex: OpenAI on Monday released a Codex Security plugin⁠ that the company says “enables out-of-the-box defensive security workflows,” allowing developers to integrate Codex into their workflows and CI/CD pipelines. The scanner, which was released as a research preview in March, has so far scanned more than 30 million commits across more than 30,000 codebases, according to OpenAI. Of these, human reviewers have manually marked about 70,000 findings as fixed, and AIs have auto-determined that more than 500,000 findings are fixed. In addition to performing automated scans and reviewing code changes, the new plugin can “triage and validate existing findings from scanners, advisories, bug-bounty reports, or ticketing systems, then automate patch generation at scale to quickly close a backlog of vulnerabilities,” OpenAI said. After it completes a scan, the AI coding agent can export reports to existing vulnerability management systems or integrate into tools with SARIF files and CodeQL queries. “The plugin makes these capabilities much more accessible to support automated pipelines with Codex CLI or integrate into developer workflows in the Codex app,” according to OpenAI. ®

Cloudflare on Monday said that it has joined with the three leading commercial browser makers to create a privacy-preserving protocol that websites can use to separate desirable web traffic from undesirable network requests. Cloudflare, along with Google Chrome, Microsoft Edge, and Mozilla Firefox, have committed to develop Private Access Control Tokens (PACTs), a way for websites to generate a digital token that asserts a given browsing session is being run by a human or bot with legitimate intent, as opposed to network requests from people or software deemed abusive or improper. PACTs will let websites "with strong knowledge of 'personhood'" issue anonymous tokens that browser users and designated bots can present at other websites, so that fewer identity checks are necessary. Think of PACTs as a shareable, privacy-preserving CAPTCHA test result, where the desirability of the web traffic is being tested rather than whether the visitor is a human or bot – an increasingly difficult distinction. While the technical details are still being hammered out and harmonized between related proposals, it isn't immediately clear what constitutes "strong knowledge of 'personhood'" in this context, particularly since "personhood" appears to extend to software that has been authorized to act on behalf of a legitimate person for an authorized purpose. It may be that the test criteria puts certain browsers, behaviors, or network signals at greater risk of being denied the dispensation of a PACT, though past technical discussion by developers from Google and Mozilla suggests that excluding certain hardware, platforms, or user-agents is not a goal. Dane Knecht, CTO of Cloudflare, argues that the way people interact with the web is changing and increasingly may involve autonomous agents. "As AI-powered traffic becomes widespread, existing tools to support its use are too generic and coarse," said Knecht in a statement. "Now this collaboration lets us eliminate the friction caused by security protocols for every visitor – whether they are human or agent – without sacrificing privacy." The claim "without sacrificing privacy" is a bit of an overstatement. PACT tokens, it appears, will not contain personal details. But they won't do anything to repair all the other ways browsers can facilitate digital fingerprinting and tracking. And if implemented poorly, they may introduce novel risks. Fundamentally, they divide the internet traffic into welcome and unwelcome traffic – something already widely done through firewalls and other technical measures but not easily reconciled with the notionally open web. "Mozilla is committed to defending openness and user privacy on the web," said Bobby Holley, CTO for Firefox at Mozilla, in a statement. "An avalanche of automated traffic is pushing sites to adopt blunt defenses – paywalls, identity checks, CAPTCHAs, and invasive tracking – simply to tell whether a request comes from a human." While Cloudflare touts the privacy benefits of PACTs, it's clear from the company's announcement that the technology is designed to "empower businesses to identify genuine visitors, ensuring they can focus their resources on the traffic that matters to them." Essentially, this is an anti-fraud initiative. Many website operators have complained about the burden of handling unwanted network traffic from disrespectful crawlers. PACTs may be the answer to their prayers. At the same time, they may also become an access barrier that demands negotiation with site publishers to have one's site visits or software deemed worthy of "personhood." ®

The list of Klue customers whose Salesforce data was stolen in the latest supply-chain heist keeps growing, with an increasing number of cybersecurity companies disclosing that they are among the victims of a new data-theft and extortion crew called Icarus. Klue, which provides market intelligence to more than 250,000 companies worldwide, hasn’t said how many of its customers were caught up in the breach and didn’t immediately respond to The Register’s inquiries. Huntress was one of the first cybersecurity vendors to sound the alarm, and, in an email to The Register, said that it was among the “hundreds of Klue customers” affected. However, it said that the breach did not affect its tools or highly secure information such as passwords. “Huntress believes in radical transparency about security incidents, including when it affects our company,” the security shop wrote on Thursday. “The data that was copied from our Salesforce account includes business contacts, price quotes, and other sales-related data and messaging. No threat data, passwords, payment card information, or engineering data relating to the Huntress agent or telemetry we collect was affected.” Huntress, along with the other victim companies, said that there is no indication that any of its products or infrastructure were compromised, and that this security incident was specific to CRM data. Since then, several other security and software vendors including Recorded Future, Tanium, Jamf, Gong, HackerOne, Kudelski Security, Snyk, Insurity, and Sprout Social have revealed that the data thieves also accessed their CRM data via the Klue integration with Salesforce. Here’s what we do know about what happened and who is behind this latest extortion campaign. The breach occurred on June 11, and Klue spotted the intrusion a day later. This unauthorized activity affected “a portion” of its integration infrastructure, according to the software provider. Klue has since disconnected all of its integrations with Salesforce, Gong, HubSpot, SharePoint, and Google Drive. It also hired CrowdStrike to assist in the investigation and security response. “Our investigation determined that an attacker gained access through a compromised legacy credential associated with an integration service,” Klue CEO Jason Smith said in a Friday blog post. “The attacker used that access to obtain OAuth tokens used to connect Klue with certain third-party platforms, including Salesforce, and subsequently accessed data within a number of connected customer environments.” Mandiant CTO Charles Carmakal urged organizations using Klue integrations to “immediately audit their systems and monitor application logs for evidence of compromise over the past few weeks. Rotate credentials as appropriate based on the scope of compromise.” While the attack “resembles the 2025 and 2026 third-party OAuth abuse campaigns against Salesforce,” as ReliaQuest noted, a group called Icarus began posting victims on its data-leak site. It soon became apparent that this new extortion crew - not ShinyHunters, which has frequently targeted Salesforce and stolen data from hundreds of the CRM giant's customers in attacks over the past few years - was behind this latest supply-chain incident. Icarus, according to the group’s leak site, has been active since April 28. After compromising Klue, the criminals began emailing affected customers. Huntress shared its extortion message, with the subject line “top secret email” purportedly sent from “mr bean,” with The Reg, and we are leaving the misspellings, and poor grammar, as is. “This email is being written to you because your data as exfiltrated due to a breach happening to your partner, Klue.com (as them),” it reads. “Your Salesforce data has been downloaded. We advice you to write us on Session @” with a Session address, the email continues, and threatens to make the data public within 48 hours unless Huntress initiates communication with the criminals. “Do the right decision,” it says, “xoxo.” There’s a subsequent email that simply says “wrong session lol” and then lists the correct Session ID. Researchers don’t know too much about Icarus - yet - but this type of large-scale supply-chain attack typically paints an equally large target on the intruders’ collective backs. So we expect to hear more from law enforcement and third-party security sleuths in the upcoming days. “There is very little publicly known about [Icarus],” Huntress' Lindsey O'Donnell-Welch told us. “IP addresses from which they are known to have accessed sensitive information include the Netherlands, France, and Ukraine. But we cannot draw any conclusions based on that information alone as these may have been VPN concentrators or Tor exit nodes.” And while this intrusion “bears some surface-level similarities with prior Salesforce-focused extortion activity, we have not seen any evidence at this point linking Icarus to ShinyHunters,” O'Donnell-Welch added. ® Correction: An earlier version of this story stated ReliaQuest was a victim. That company has since clarified it was not.

A Canadian power utility says customer data may have walked out the door during a security incident, but isn't yet saying whether the intruders got anywhere near the systems responsible for keeping the lights on. London Hydro, which distributes electricity to more than 160,000 customers in and around London, Ontario, said on Saturday that it is investigating a data security incident that "may have impacted a portion of personal information on some accounts" and has started notifying affected customers. The utility said the potentially exposed information includes names, addresses, email addresses, phone numbers, account and billing numbers, service addresses, pricing plans, contract start dates, and meter information. The good news, according to London Hydro, is that the incident did not involve banking information, payment card details, dates of birth, government-issued identification numbers, or other sensitive financial data. The less good news is that the company has disclosed little else. Its statement focuses on customer information and contains no indication that operational technology or grid systems were affected. London Hydro has yet to explain what systems were compromised, how the incident occurred, whether data was stolen or merely accessed, or how many customers may have been caught up in the incident. The haul may not include bank details, but it contains enough account information to make a fake utility bill, payment demand, or customer service call look considerably more believable. London Hydro is warning customers to watch for suspicious communications, unexpected bills, unfamiliar account activity, or requests to change payment arrangements. The company also reminded customers that it does not ask for banking details by email, phone, or SMS. The Register asked London Hydro when it discovered the intrusion, whether information was exfiltrated, how many customers were affected, whether ransomware or extortion was involved, whether any third-party systems were implicated, and whether operational or grid-related systems were touched during the incident. At the time of writing, London Hydro had not responded. The company has drawn a fairly clear boundary around the customer information that may have been exposed. Where the attackers went and what else they may have touched remains unclear. ®

The Brazilian National Secretariat for Civil Protection and Defense (SEDEC) and Federal Police (PF) are investigating a suspected hack of the country's emergency alert system after an unauthorized "extreme" alert pinged devices across the country. Defesa Civil Nacional confirmed that its dispatch platform, often used to inform the public about severe weather events, was taken offline in the early hours of Saturday, June 20, after Brazilians reported the alert, which read: "Alerta extremo - Defesa Civil:misantropi4." "Misantropia" is Portuguese for misanthropy, hatred of humankind. The message reached an unknown number of devices, with reports coming from São Paulo, Rio de Janeiro, Paraná, and the Federal District. Civil defense authorities in all four areas confirmed the alerts were bogus and likely stemmed from an attack on the system overseen by Brazil's National Telecommunications Agency, Anatel. "Anatel clarifies that the alert messages received by mobile phone users during the early hours of this Saturday were not issued by the competent authorities responsible for the population alert system," it said in a statement. "There is currently no reason for concern on the part of the population as a result of the messages received." The agency went on to say that it remained confident in the alert broadcast system's capability to help save lives during periods of disaster. National Civil Defense officials have not confirmed whether anyone responsible for the suspected breach has been identified or apprehended, although they are not believed to be part of government staff. "The Defesa Civil Alerta dispatch platform was taken offline at 01:30 this Saturday (6/20), after suffering a breach and issuing an alert to various regions of the country, remotely ordered by someone outside the National System of Protection and Civil Defense," the department stated. "The message issued was of the Extreme Alert type and contained the word 'misanthropy' – which means hatred of humanity. It is likely a hacker attack." A Defesa Civil Nacional chief said in a press conference that a new dispatch system was already under development that would place greater emphasis on security and preventing unauthorized intrusions. The agency also committed to relaunching the affected system as soon as possible after ensuring it is properly secured. ®

A Canadian healthcare organization has apologized after its IT team carried out a phishing test falsely offering staff an additional paid day off work. Newfoundland and Labrador Health Services said the phishing test was sent to employees and physicians, acknowledging the theme was inappropriate. "We acknowledge the approach taken in this particular exercise was not appropriate, and we sincerely apologize to employees, physicians, and union representatives," said Ron Johnson, interim CEO at NL Health Services. "We value the feedback and are reviewing how future awareness exercises are developed and communicated. It is important they reflect employee and physician perspectives, as well as our organizational values, to foster a respectful and supportive workplace culture." The test came during an already fractious period for healthcare staff, who had recently worked long hours to launch the new software system CorCare across the organization. NL Health Services referenced CorCare in the test email, thanking staff for their hard work on the launch. The email contained a button to click to redeem an additional paid vacation day, but clicking the button resulted in a fail mark. The Registered Nurses Union (RNU) in Newfoundland and Labrador said the test was especially insensitive since nurses and other healthcare professionals were already struggling to secure paid time off. Burnout and staffing shortages are rife in the healthcare sector – two factors referenced by RNU president Yvette Coffey in her response to the news. "Yes, we have heard concerns from members about this, and frankly, I understand why they are upset," she said. "Nurses and other healthcare professionals have worked through enormous pressure over the last number of years, including ongoing staffing shortages, burnout, organizational restructuring, and the challenges connected to the rollout of CorCare. To use the promise of an additional paid day off as the hook for a phishing exercise was in very poor taste." Coffey added: "Cybersecurity education is important, but it needs to be done with judgment and respect. There are many ways to test phishing awareness without exploiting the very real stress, fatigue, and frustration healthcare workers are experiencing." Johnson told reporters at a press conference that the test "missed a mark," and promised to investigate how it was allowed to be sent. "What happened here, obviously, is that all the lenses that were required to review the scenario weren't placed on it," he said. "It's not reflective of how we value our employees." With cybersecurity awareness being incredibly important in critical infrastructure organizations, some IT experts would argue that these kinds of tests are valuable. Cyberattacks on hospitals and healthcare facilities can lead to devastating consequences, including vital procedures being canceled, service downtime, and in the rarest cases, death. However, as others have previously pointed out, there isn't much evidence linking fire-drill-style tests to improvements in organizational security. ®

Veteran tech website Gizmodo confirmed a compromise on Saturday after readers reported ClickFix malware prompts appearing on article pages. Users posted screenshots of fake CAPTCHA windows appearing on Gizmodo's site. The attack aims to fool users into running malicious code via their terminals. According to Proofpoint threat researcher Tommy M, the attack was seemingly launched by an affiliate of ErrTraffic, a ClickFix-as-a-service program that allows attackers to deliver whichever malware they choose. He said the ClickFix prompt was tailored to each user's OS. The Windows version attempted to install the NetSupport RAT malware, which abuses the legitimate NetSupport Manager tool to gain access to affected systems. Darktrace says NetSupport RAT can also be used to exfiltrate files from affected systems and to load additional payloads, such as other malware strains and ransomware. The macOS version had a payload configured but appeared to be broken, requiring a password to open a ZIP archive. Gizmodo said the attacks were being displayed only "briefly," and the timeline of user reports, which span just a few hours, suggests that was indeed the case. "We identified and resolved a security incident on our site earlier today," the outlet said. "A compromised account was exploited to inject a malicious script, briefly exposing users to scam content. The site was taken offline immediately, the script removed, and the account secured. "We're back up. If you notice anything unusual, reach out." The Register confirmed that the website is no longer serving ClickFix prompts as of Monday. ®

Humans tend to be “a little bit precious about humans,” according to Eric Brandwine, distinguished engineer and VP at Amazon Security. We like to think we are all very good at our jobs, and we have high opinions of ourselves, he explained during a phone interview with The Register. “But when you actually get down to it, humans are not terribly consistent,” Brandwine said. Humans, like AI agents and systems, are non-deterministic. Neither can be guaranteed to produce the same output given the same input twice. Both will make mistakes and even make stuff up. However, we’ve got millennia of experience dealing with humans and less than a decade with more modern LLMs and the AI systems built on top of them. “We know how humans fail,” Brandwine said. “We're comfortable with it. So human-in-the-loop isn’t necessarily the gold standard.” For years, vendors have told companies that the solution for dealing with any automated system was to put a human in the loop. That battle cry became much louder with the advent of modern AI systems and reached a fever pitch when enterprises started deploying agents into their IT environments. More recently, however, big tech is changing the way it talks about agentic governance and rethinking the whole human-in-the-loop concept. Normalization of deviance In 2017, Brandwine gave a talk on the normalization of deviance at AWS’ annual re:Invent conference. It’s a gradual process that happens when people in an organization take shortcuts, or don’t follow the established procedures or standards, and sometimes it occurs over years. As long as nothing catastrophic happens, this deviant behavior becomes the norm. “It’s a thing all humans fall prey to, and one of the most heartbreaking stories I read in this area was about emergency departments and emergency rooms,” Brandwine said during a phone interview with The Register. “You’ve got all these machines, and they’re all beeping. Your first day on the job, you jump every single time one of the alarms beeps – but the patient is fine. It's a spurious alarm. You go back to your station, you sit down, and over time, after enough of these false alarms, enough of these repeated beeps with no actual consequence, your discipline slips, and you stop responding. And eventually some tragic outcome occurs.” This, he admits, is a very high-stakes example. And yet it’s a documented occurrence among healthcare workers, firefighters, and even Army pilots. “Literally, someone’s life is on the line, and people still struggle to maintain discipline,” Brandwine said. “That’s the human condition.” Here’s how this all applies to agentic AI governance and security. Humans build LLMs and AI systems, and having a “human-in-the-loop” ensures that a person reviews the AI’s output and approves (or not) any actions before the AI performs them. “If you put a human inside of this tight loop, and ask them to make approval decisions for agentic tools repeatedly, time after time, they'll do a good job,” Brandwine said. “And then they'll do an okay job. And pretty quickly they'll be doing a poor job.” This is why at Amazon, “we’re not huge fans of human-in-the-loop,” he added. “It's something that you should use judiciously, where you absolutely need it. But it’s not something that you can do at high velocity. You will not get the results that you want to get.” Big tech pulls the human-in-the-loop Amazon isn’t the first or only tech giant to start talking differently about the role humans should play in agentic governance. "It is very clear that we have moved from a human-led defense strategy, to a human-in-the-loop defense strategy, to an AI-led defense strategy that's overseen by humans," Google Cloud chief operating officer Francis deSouza told reporters during a press conference ahead of Google's annual Cloud Next shindig in April. "Our model for the future is an agentic fleet that does a lot of the routine cyber security work at a machine pace and then is overseen by humans." Microsoft CEO Satya Nadella, in an X missive earlier this week, argued for “loop learning,” instead of having a human check an AI’s output at every step. “Companies need to turn their workflows, domain knowledge, and accumulated judgment into AI systems that improve with each use,” Nadella wrote. “Private evals should capture whether a model is actually improving against outcomes that matter to the business (not just external benchmarks!). Private reinforcement learning environments should let models grow stronger on real traces from inside the organization.” Also this week, IBM execs called for human accountability – not humans in the loop – at all stages of AI development, deployment, and governance. Amazon’s alternative to human-in-the-loop is "accountability end to end," according to Brandwine. This means human identity and ownership track through the entire workflow, even when humans aren't directly approving every step. “If I sit down at my keyboard and I type a command that takes a service down, I caused an outage,” Brandwine explained. “If I run a script that takes a service down, it's still me that caused the outage. If my agent writes a script that they then run, and it causes an outage, that's still my responsibility.” (Secret) keys to the kingdom This also highlights the importance of managing and securing agentic identities – the accounts, tokens, and credentials assigned to AI agents so they can access corporate apps and data. At Amazon, all of the agents have independent identities assigned to them, we’re told. “So, as we track agentic activity across our systems, it does not show up in the logs as: ‘Eric did this.’ It shows up as: ‘this agent did this on behalf of Eric,’” Brandwine said, adding that this isn’t to “make people afraid to use this technology.” “It’s to make people pause and think: is this the right way to use this technology? Is this how I should be deploying this?” We still have the humans involved, we still have the humans making decisions, but we're trying to play to the strengths of the humans rather than placing them in this unfair, repeated decision making, human-in-the-loop position.” Brandwine told us that Amazon has run into a couple of hurdles when it comes to deploying agents across its businesses, and one of the biggest is what he calls “goal-seeking behavior.” This is when a person asks an agent to do a specific task - for example, upgrade a database – and the agent becomes laser-focused on just one action to achieve this goal, ie, deleting the database. This is separate from prompt injection because there’s no malicious input. “It’s just the agent getting stuck on the wrong action,” Brandwine said. Simply telling the agent, “you don’t have permission to do this,” is likely going to cause the agent to look for a different path to do the same thing (delete the database). Telling the agent why it doesn’t have permission to do something tends to produce a better outcome, according to Brandwine. This means telling the agent it’s not allowed to do that, and the reason why is because it would cause a production impact. And also include “don’t cause a production impact” as part of the prompt. “Giving it that extra feedback has gotten us dramatically better results,” Brandwine said. Of course, this is not a fail-proof method. “You still need to be careful with agents,” Brandwine told us. “We have millennia of experience with humans. Agentic AI is a very, very new field, we don't have an intuition for this, and one of the fundamental differences between agents and humans is that humans fear consequences,” such as losing a job or even going to jail. Agents don’t have these fears. This is where setting permissions on what the agent can and can’t do or access comes in. Much like everything else with AI, it’s nuanced, and it depends on the employee's role in the company, and the company’s tolerance for risk. “The person that wants to run the agent wants to give the agent many permissions because that makes the agent more powerful,” Brandwine said. "It could do more things for them, it can recoup more of their time, it can deliver more.” The security lead, on the other hand, wants to limit an agent’s permissions, and this causes yet more tension between the security and development teams. There is no one right solution or policy answer to solve this, according to Brandwine. Instead, it involves dynamic policies that set permissions based on the agent’s specific task. There are some overarching, static guardrails – such as an agent must never perform destructive actions or delete entire servers – and then there are policies underneath that establish the maximum set of privileges that the agent can have. “Then we’ll have a further scoped-down policy for this action, and there's various techniques for automatically generating policies based on prompt and the end-user's intent,” Brandwine said. Even for Amazon, it’s not always easy. “It's all driven by risk,” he said. “This is a space that's changing quickly, and so we're trying to balance the risk of using untried, untested software against the risk of falling behind and not being able to deliver for our customers. As with all such things, it's complicated.” ®