Logos Dojo No. 116

執筆者:

カテゴリ:

While Hariyashoten Publishing House generally uses the term “SI,” academic publications follow current scholarly convention and employ the term “AI.

Logos Dojo No. 116

Appropriate Use of AI Agents in Information Security in the Generative AI Era

— Defining the Boundary Between Automation and Human Oversight —

1. Introduction

As generative AI continues to advance, its use for defensive purposes in information security is becoming increasingly important.

Current generative AI systems can analyze large volumes of logs, source code, configuration files, network records, and vulnerability information within a relatively short period of time. They can therefore help identify anomalies and potential vulnerabilities that might otherwise require considerable human effort to detect.

AI agents are also moving beyond text generation. They can increasingly connect to operating systems, servers, cloud environments, databases, development tools, and other systems, and can perform certain operations with varying degrees of autonomy.

These capabilities create substantial opportunities for information security, but they also introduce new risks.

Current AI systems are not fully reliable. When the available information is incomplete or when complex context is misunderstood, an AI system may produce incorrect reasoning, false positives, false negatives, or other errors.

If an AI agent has direct access to operational information systems, such errors may lead to a sequence such as:

For this reason, the key issue in information security in the generative AI era is not simply:

Rather, the more important question is:

The central issue is therefore one of system design and authority allocation.

2. Areas in Which AI Can Be Used Proactively

At the current stage of development, AI is generally most suitable for tasks involving:

Possible uses include:

  • identifying potential anomalies in server and network logs;
  • detecting possible vulnerabilities in source code;
  • reviewing potentially unsafe constructs in SQL, JavaScript, PHP, Python, and other languages;
  • checking software and dependency versions;
  • comparing published vulnerability information with an organization’s system configuration;
  • assessing the possible impact of security patches;
  • summarizing large volumes of logs and alerts during an incident;
  • generating test code;
  • proposing code fixes or configuration changes;
  • identifying possible causes of system failures; and
  • drafting recovery procedures and response plans.

These tasks share an important characteristic:

For example, if an AI system reports that a particular software library may contain a vulnerability, the report itself does not alter the system.

A human engineer can review the finding, compare it with vulnerability information and source code, and, if necessary, test a correction in a non-production environment.

In such a workflow, even if the AI makes an imperfect judgment, the probability that the error will immediately result in a major production failure is relatively low.

At the current stage of technology, it is therefore reasonable to view AI primarily as:

within information security operations.

3. Areas in Which AI Agent Errors May Create Significant Risk

Greater caution is required when an AI agent is allowed to make decisions and directly modify a production environment.

Examples include:

  • modifying production server configurations;
  • changing firewall rules;
  • modifying network routes;
  • updating or deleting database records;
  • changing user accounts or permissions;
  • automatically rewriting production source code;
  • immediately applying security patches;
  • stopping or restarting servers and services;
  • changing cryptographic keys or authentication credentials;
  • deleting backups; and
  • automatically isolating production networks.

In these situations, a chain of events may occur:

Depending on the system involved, the impact may extend beyond a technical failure:

Therefore, the level of operational authority granted to an AI agent should not be determined solely by the technical capability of the AI.

It should also be determined by the potential consequences of an erroneous action.

4. Separating Read and Write Privileges

One useful principle for the safe deployment of AI agents is to separate:

from:

In other words, the authority to obtain and analyze information should not automatically imply the authority to modify the system.

For example:

By contrast:

This principle can be summarized as follows:

This approach does not reject the capabilities of AI agents.

Rather, it seeks to make active use of their analytical speed while limiting the potential consequences of erroneous actions.

5. Reversibility as a Criterion for AI Authority

When an AI agent is allowed to make changes, another important criterion is whether the action can be reversed.

Examples of relatively reversible operations include:

  • restarting a test environment;
  • creating temporary virtual machines or containers;
  • updating a staging environment;
  • modifying test data; and
  • adding temporary monitoring configurations.

By contrast, operations such as the following may be difficult or impossible to reverse:

  • permanently deleting a production database;
  • deleting backups;
  • deleting cryptographic keys;
  • overwriting large volumes of production data;
  • deleting critical user accounts; and
  • modifying the recovery path itself.

Conceptually, the relationship can be expressed as:

In other words, highly reversible actions are generally easier to delegate to AI, while irreversible actions require stronger human oversight.

6. Risk-Based Levels of AI Use

The use of AI agents need not be classified only as either permitted or prohibited.

A more practical approach is to define levels of AI authority according to risk.

AI UseExampleGeneral Treatment
Information analysisAnalysis of logs, network records, and configurationsSuitable for proactive use
Code analysisDetection of possible vulnerabilitiesSuitable for proactive use
Information organizationOrganizing large numbers of alerts and vulnerability reportsSuitable for proactive use
Remediation proposalsGenerating code or configuration changesGenerally suitable
Test generationCreating automated testsGenerally suitable
Test-environment changesUpdating staging systemsMay be automated under defined conditions
Production configuration changesFirewall, OS, or web server changesHuman approval should be required
Production data changesDatabase updates or deletionsHigh risk
Service shutdownStopping operational systemsGenerally unsuitable for AI-only execution
Backup or key deletionActions that may eliminate recovery capabilityShould not be delegated to AI alone

The objective is not:

but rather:

7. Human-in-the-Loop Operations

When AI agents are introduced into information security operations, humans should not necessarily be removed from the decision-making process.

A practical workflow may be:

For more critical operations, an additional approval layer may be appropriate:

In this model, AI does not replace human judgment.

Instead, it serves as:

This approach combines the strengths of generative AI in large-scale information analysis with human responsibility for high-impact decisions.

8. AI Agents Must Also Be Treated as Security Assets

Another important consideration is that AI agents should not be viewed only as security tools.

The AI agent itself must also be treated as an object of security control.

If an AI agent can access servers, cloud platforms, source code, databases, or other critical resources, compromise of its credentials or operating privileges could turn the agent itself into an attack path.

Appropriate measures may therefore include:

  • using dedicated accounts for AI agents;
  • applying the principle of least privilege;
  • avoiding direct embedding of authentication secrets in the model;
  • retaining detailed operation logs;
  • requiring approval for high-impact operations;
  • limiting the commands and APIs available to the AI;
  • separating production and test-environment privileges; and
  • implementing mechanisms that automatically stop abnormal operations.

Information security therefore involves not only:

but also:

9. AI Automation Must Be Stoppable

When autonomous AI agents are introduced, safe operation requires not only the ability to perform tasks correctly but also the ability to stop abnormal behavior.

For example, if an AI agent begins making a large number of configuration changes within a short period of time, the system should be able to apply controls such as:

  • limits on the number of operations;
  • restrictions on the scope of change;
  • rate limits on execution;
  • automatic suspension when abnormal activity is detected; and
  • an emergency stop controlled by a human operator.

This means that the design requirements for information systems should include not only:

but also:

It is therefore insufficient to assume that more capable AI agents should automatically receive broader privileges.

A more appropriate principle is:

10. Conclusion

In the generative AI era, attackers may increasingly use AI to accelerate source-code analysis, vulnerability discovery, and information gathering.

For this reason, defensive organizations can also benefit from using AI to accelerate:

However, current AI agents remain capable of incorrect judgments and unintended actions. It is therefore not always appropriate to fully automate all information-security operations.

At the current stage of technology, the following principles are reasonable:

  1. Use AI actively for analysis, detection, and recommendation.
  2. Require human approval for significant production changes.
  3. Grant AI agents only the minimum privileges required.
  4. Design changes to be reversible whenever possible.
  5. Apply stronger restrictions to irreversible operations.
  6. Separate test environments from production environments.
  7. Treat AI agents themselves as objects of security management.
  8. Maintain the ability for humans to stop and control autonomous AI operations.

Accordingly, when AI agents are introduced into information security, it is useful to maintain the distinction:

AI should not initially be treated as a fully autonomous administrator. A more cautious approach is to use AI first as a high-speed analytical and decision-support system, and then gradually expand the scope of autonomy according to the criticality of the system, the reversibility of operations, and the potential scale of harm.

Information security in the generative AI era does not require either excessive fear of AI or unrestricted delegation of authority to AI.

The more important objective is to combine:

Designing the relationship among these three elements may become one of the fundamental principles of information security as AI agents become more widely integrated into information-processing systems.

Note on the Preparation and Licensing of This Essay

Most of the text in this essay was generated with ChatGPT, a representative large language model–based generative AI system.

The present text was developed through an iterative dialogue that began with an initial draft focused on a specific industry and was subsequently expanded and revised into a more general discussion.

For this reason, this installment of Logos Dojo (No. 116) is released under CC0 1.0 Universal, with the intention of dedicating the text to the public domain to the extent permitted by law.

License: CC0 1.0 Universal (Public Domain Dedication)

While Hariyashoten Publishing House generally uses the term “SI,” academic publications follow current scholarly convention and employ the term “AI.