20250SB__083396AMD INTRODUCED 2025-02-21 AMENDED_SENATE 2025-03-26 AMENDED_ASSEMBLY 2025-07-07 AMENDED_ASSEMBLY 2025-07-17 2025 SB AMD Introduced by Senator McNerney LEAD_AUTHOR SENATE McNerney

An act to add

Article 6.6 (commencing with

Section 8954.50) to

Chapter of Division of Title of the Government Code, relating to state government. state government Critical infrastructure: artificial intelligence systems: human oversight. Existing law, the California Emergency Services Act, establishes the California Cybersecurity Integration Center within the Office of Emergency Services to serve as the central organizing hub of state government’s cybersecurity activities and to coordinate information sharing with various entities.

Existing law also requires the Technology Recovery Plan element of the State Administrative Manual to ensure the inclusion of cybersecurity strategy incident response standards for each state agency to secure its critical infrastructure controls and information, as prescribed.

This bill would require, on or before July 1, 2026, an operator, defined as a state agency responsible for operating, managing, overseeing, or controlling access to critical infrastructure, that deploys a covered artificial intelligence (AI) system, as defined, to establish a human oversight mechanism that ensures a human monitors the system’s operations in real time and reviews and approves any plan or action proposed by the covered AI system before execution, except as provided.

The bill would require the Department of Technology to develop specialized training in AI safety protocols and risk management techniques to oversight personnel. The bill would require oversight personnel for an operator to conduct an annual assessment of its covered AI systems, as specified, and to submit a

summary of the findings to the department. The bill would make findings and declarations related to its provisions. This bill would require any entity that engages in conduct that could materially impact critical infrastructure safety, security, or operations to report an artificial intelligence (AI) adverse event, as defined, in a form and manner prescribed by the Office of Emergency Services, as provided. The bill would subject each entity that fails to provide an AI adverse event report to a specified civil penalty.

The bill would authorize the office to, among other things, authorize public or private entities to receive information about individual events or aggregated statistics for the purpose of collaboratively addressing identified harms, except as provided. The bill would include related findings and declarations.</xhtml:p>"?> The bill would preclude disclosure of specified information by the office.

Existing constitutional provisions require that a statute that limits the right of access to the meetings of public bodies or the writings of public officials and agencies be adopted with findings demonstrating the interest protected by the limitation and the need for protecting that interest. This bill would make legislative findings to that effect. MAJORITY NO YES NO NO NO NO NO NO NO NO The people of the State of California do enact as follows:

SECTION 1. The Legislature finds and declares all of the following: (

a) In response to the rapid advancement of generative artificial intelligence (GenAI) and its growing integration across public and private sectors, the Governor issued

Executive Order No. N-12-23, which established a comprehensive policy framework to responsibly explore and govern the deployment of GenAI systems within the state. (

b) The Governor convened the Joint California Policy Working Group on AI Frontier Models to evaluate the potential risks and governance needs associated with the deployment of powerful artificial intelligence models. (c)<xhtml:span class="EnSpace"/>The working group’s report recommends the establishment of adverse event reporting mechanisms for artificial intelligence systems, especially in high-risk domains such as critical infrastructure, to ensure transparency, accountability, and public safety.</xhtml:p><xhtml:p>(d)<xhtml:span class="EnSpace"/>The report underscores the importance of centralized, timely, and clear reporting of AI malfunctions or failures that could lead to physical harm, data breaches, or disruption of essential services.

These recommendations are echoed by expert consensus on artificial intelligence safety protocols nationally.</xhtml:p><xhtml:p>(e)</xhtml:p>"?> (

c) The Governor’s executive order also emphasized the urgent need for workforce development and training to ensure that public sector employees have the technical expertise and practical tools necessary to oversee and manage artificial intelligence systems safely and effectively. (f)<xhtml:span class="EnSpace"/>As California integrates artificial intelligence into critical infrastructure operations, it is essential that state agencies implement human verification mechanisms, conduct rigorous risk assessments, and comply with mandatory adverse event reporting standards to protect public health, safety, and the integrity of essential systems.</xhtml:p>"?> <caml:Num>SEC. 2.</caml:Num><caml:ActionLine action="IS_ADDED" xlink:href="urn:caml:codes:GOV:caml#xpointer(%2Fcaml%3ALawDoc%2Fcaml%3ACode%2F%2Fcaml%3ALawSection%5Bcaml%3ANum%3D'8592.51'%5D)" xlink:label="fractionType: LAW_SECTION" xlink:type="locator">Section 8592.51 is added to the <caml:DocName>Government Code</caml:DocName>, <caml:Positioning>immediately following

Section 8592.50</caml:Positioning>, to read:</caml:ActionLine><caml:Fragment><caml:LawSection id="id_4E41FD78-A0E5-45DF-BD7D-0F0D3F8A67F4"><caml:Num>8592.51.</caml:Num><caml:LawSectionVersion id="id_EFB87B8A-6E48-403E-B534-5404C5A76AEE"><caml:Content><xhtml:p>(a)<xhtml:span class="EnSpace"/>For purposes of this section, the following

definitions apply:</xhtml:p><xhtml:p>(1)<xhtml:span class="EnSpace"/>“AI adverse event” means an incident, circumstance, or series of events where the development, deployment, use, or malfunction of an artificial intelligence (AI) system or automated decision system in critical infrastructure has caused or contributed to any of the following:</xhtml:p><xhtml:p>(A)<xhtml:span class="EnSpace"/>Death of any person.</xhtml:p><xhtml:p>(B)<xhtml:span class="EnSpace"/>Serious physical injury requiring medical treatment.</xhtml:p><xhtml:p>(C)<xhtml:span class="EnSpace"/>Significant disruption to critical infrastructure operations lasting more than one hour.</xhtml:p><xhtml:p>(D)<xhtml:span class="EnSpace"/>Unauthorized access to or compromise of sensitive data affecting more than individuals.</xhtml:p><xhtml:p>(E)<xhtml:span class="EnSpace"/>Material financial loss greater than fifty thousand dollars ($50,000) to any person or entity. </xhtml:p><xhtml:p>(F)<xhtml:span class="EnSpace"/>System failure that requires manual intervention to prevent harm or restore service.</xhtml:p><xhtml:p>(G)<xhtml:span class="EnSpace"/>Any failure of the AI system to perform as intended that could reasonably lead to mass casualty events or widespread critical infrastructure failure.</xhtml:p><xhtml:p>(2)<xhtml:span class="EnSpace"/>“Artificial intelligence” (“AI”) means an engineered or machine-based system that varies in its level of autonomy that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.</xhtml:p><xhtml:p>(3)<xhtml:span class="EnSpace"/>“Automated decision system” means a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decisionmaking and materially impacts natural persons. “Automated decision system” does not include a spam email filter, firewall, antivirus software, identity and access management tools, or a calculator.</xhtml:p><xhtml:p>(4)<xhtml:span class="EnSpace"/>“Critical infrastructure” means systems or assets so vital to the state that the incapacity or destruction of those networks, systems, or assets would have a debilitating impact on public health, safety, economic security, or any combination thereof, including any of the following infrastructure:</xhtml:p><xhtml:p>(A)<xhtml:span class="EnSpace"/>Transportation.</xhtml:p><xhtml:p>(B)<xhtml:span class="EnSpace"/>Energy.</xhtml:p><xhtml:p>(C)<xhtml:span class="EnSpace"/>Food and agriculture.</xhtml:p><xhtml:p>(D)<xhtml:span class="EnSpace"/>Communications.</xhtml:p><xhtml:p>(E)<xhtml:span class="EnSpace"/>Emergency services.</xhtml:p><xhtml:p>(F)<xhtml:span class="EnSpace"/>Financial services.</xhtml:p><xhtml:p>(5)<xhtml:span class="EnSpace"/>“Operator” means a state agency in charge of critical infrastructure.</xhtml:p><xhtml:p>(b)<xhtml:span class="EnSpace"/>(1)<xhtml:span class="EnSpace"/>An operator deploying artificial intelligence that could materially impact critical infrastructure safety, security, or operations shall establish a human oversight mechanism to do both of the following:</xhtml:p><xhtml:p>(A)<xhtml:span class="EnSpace"/>Monitor the artificial intelligence system’s operations in real time.</xhtml:p><xhtml:p>(B)<xhtml:span class="EnSpace"/>Review and approve any plan or action proposed by an artificial intelligence system before execution.</xhtml:p><xhtml:p>(2)<xhtml:span class="EnSpace"/>This subdivision shall not apply to an existing automated decision system that is critical to state infrastructure if the required human oversight would cause an immediate pause that would destabilize that system.</xhtml:p><xhtml:p>(c)<xhtml:span class="EnSpace"/>The Department of Technology shall administer specialized training in artificial intelligence safety protocols and risk management techniques to be given to oversight personnel.</xhtml:p><xhtml:p>(d)<xhtml:span class="EnSpace"/>(1)<xhtml:span class="EnSpace"/>An operator shall conduct an annual assessment of its artificial intelligence systems and automated decision systems that does all of the following:</xhtml:p><xhtml:p>(A)<xhtml:span class="EnSpace"/>Evaluates compliance with this section.</xhtml:p><xhtml:p>(B)<xhtml:span class="EnSpace"/>Evaluates system performance and safety.</xhtml:p><xhtml:p>(C)<xhtml:span class="EnSpace"/>Identifies and evaluates potential risks and vulnerabilities, including those that could lead to mass casualty events.</xhtml:p><xhtml:p>(2)<xhtml:span class="EnSpace"/>An operator shall submit a

summary of the assessment findings to the Department of Technology.</xhtml:p><xhtml:p>(3)<xhtml:span class="EnSpace"/>The assessment shall coincide with any requirement on the operator to perform a risk analysis pursuant to subdivision (

b) of

Section 11549.65.</xhtml:p></caml:Content></caml:LawSectionVersion></caml:LawSection></caml:Fragment></caml:BillSection><caml:BillSection id="id_9A1DED25-0BC7-4BF0-A10F-685D6F7823ED"><caml:Num>SEC. 3.</caml:Num><caml:ActionLine action="IS_ADDED" xlink:href="urn:caml:codes:GOV:caml#xpointer(%2Fcaml%3ALawDoc%2Fcaml%3ACode%2F%2Fcaml%3ALawSection%5Bcaml%3ANum%3D'8592.52'%5D)" xlink:label="fractionType: LAW_SECTION" xlink:type="locator">Section 8592.52 is added to the <caml:DocName>Government Code</caml:DocName>, to read:</caml:ActionLine><caml:Fragment><caml:LawSection id="id_407B3AC1-CB36-4434-9FCC-477D0A9F2955"><caml:Num>8592.52.</caml:Num><caml:LawSectionVersion id="id_ED1B8643-3FC1-49FA-9654-BF34F7677DD3"><caml:Content><xhtml:p>(a)<xhtml:span class="EnSpace"/>Any entity that engages in conduct that could materially impact critical infrastructure safety, security, or operations shall report an AI adverse event in a form and manner prescribed by the office as follows:</xhtml:p><xhtml:p>(1)<xhtml:span class="EnSpace"/>Within four hours after detection of an AI adverse event that poses an ongoing urgent threat to public health or safety.</xhtml:p><xhtml:p>(2)<xhtml:span class="EnSpace"/>Within hours after detection of an AI adverse event involving death or serious physical injury.</xhtml:p><xhtml:p>(3)<xhtml:span class="EnSpace"/>Within hours after detection of an adverse AI event involving significant critical infrastructure disruption or data compromise.</xhtml:p><xhtml:p>(4)<xhtml:span class="EnSpace"/>Within calendar days after detection of any other AI adverse event.</xhtml:p><xhtml:p>(b)<xhtml:span class="EnSpace"/>For purposes of this section, detection occurs on the first business day that the AI adverse event is known to the operator or, by exercising reasonable diligence, should have been known to the operator.</xhtml:p><xhtml:p>(c)<xhtml:span class="EnSpace"/>An AI adverse event report shall include, to the extent known or reasonably ascertainable, all of the following:</xhtml:p><xhtml:p>(1)<xhtml:span class="EnSpace"/>A description of the artificial intelligence system or automated decision system, including, but not limited to, all of the following:</xhtml:p><xhtml:p>(A)<xhtml:span class="EnSpace"/>System type, purpose, and intended use.</xhtml:p><xhtml:p>(B)<xhtml:span class="EnSpace"/>Vendor information and version numbers.</xhtml:p><xhtml:p>(C)<xhtml:span class="EnSpace"/>Deployment environment and integration details.</xhtml:p><xhtml:p>(D)<xhtml:span class="EnSpace"/>Training data sources relevant to the incident.</xhtml:p><xhtml:p>(E)<xhtml:span class="EnSpace"/>Status of human oversight mechanisms at the time of the incident.</xhtml:p><xhtml:p>(2)<xhtml:span class="EnSpace"/>Details of the adverse event, including, but not limited to, all of the following:</xhtml:p><xhtml:p>(A)<xhtml:span class="EnSpace"/>Date, time, and location of the occurrence.</xhtml:p><xhtml:p>(B)<xhtml:span class="EnSpace"/>Circumstances leading to the event.</xhtml:p><xhtml:p>(C)<xhtml:span class="EnSpace"/>Nature and extent of the harm, damage, or disruption.</xhtml:p><xhtml:p>(D)<xhtml:span class="EnSpace"/>Number of individuals or systems affected.</xhtml:p><xhtml:p>(E)<xhtml:span class="EnSpace"/>Root-cause analysis, when available.</xhtml:p><xhtml:p>(F)<xhtml:span class="EnSpace"/>Effectiveness of human oversight mechanisms in detecting or preventing the incident.</xhtml:p><xhtml:p>(3)<xhtml:span class="EnSpace"/>Response and mitigation measures taken or being planed, including, but not limited to, any modifications to human oversight mechanisms.</xhtml:p><xhtml:p>(4)<xhtml:span class="EnSpace"/>Contact information for a designated representative of the operator.</xhtml:p><xhtml:p>(5)<xhtml:span class="EnSpace"/>Any additional information requested by the department.</xhtml:p><xhtml:p>(d)<xhtml:span class="EnSpace"/>Each entity shall be subject to a civil penalty not to exceed five hundred dollars ($500) for each seven days that the entity fails to provide an AI adverse event report required under this section.</xhtml:p><xhtml:p>(e)<xhtml:span class="EnSpace"/>The office may do any of the following:</xhtml:p><xhtml:p>(1)<xhtml:span class="EnSpace"/>Authorize entities that are not operators to voluntarily participate in the reporting system for the purpose of maximally growing the evidence base.</xhtml:p><xhtml:p>(2)<xhtml:span class="EnSpace"/>Authorize public or private entities to receive information about individual events or aggregated statistics for the purpose of collaboratively addressing identified harms.</xhtml:p><xhtml:p>(3)<xhtml:span class="EnSpace"/>Post on its internet website information about individual events or aggregated statistics.</xhtml:p><xhtml:p>(f)<xhtml:span class="EnSpace"/>The office shall not disclose any record or information within a record of the office related to enforcement of this

section that is privileged, protected by copyright, or otherwise prohibited by law from being disclosed; that is exempt from disclosure to the public under express provisions of the California Public Records Act (Division 10 (commencing with

Section 7920.000) of Title 1); or in which, based on the facts of the particular case, the public interest served by not disclosing the record clearly outweighs the public interest served by disclosure of the record.</xhtml:p></caml:Content></caml:LawSectionVersion></caml:LawSection></caml:Fragment></caml:BillSection>"?>

SEC.

Article 6.6 (commencing with

Section 8954.50) is added to

Chapter of Division of Title of the Government Code , to read: 6.6. Artificial Intelligence and Critical Infrastructure 8954.50. For purposes of this section, the following

definitions apply: (a) “Artificial intelligence” (“AI”) means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments. (b) “Automated decision system” means a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decisionmaking and materially impacts natural persons. “Automated decision system” does not include a spam email filter, firewall, antivirus software, identity and access management tools, or a calculator. (c) “Covered AI system” means an AI system or automated decision system that an operator uses to operate, manage, oversee, or control access to critical infrastructure. (d) “Critical infrastructure” means systems or assets so vital to the state that the incapacity or destruction of those networks, systems, or assets would have a debilitating impact on public health, safety, economic security, or any combination thereof, but not unintended use, including, but not limited to, the following sectors: chemical, commercial facilities, communications, critical manufacturing, dams, defense industrial base, emergency services, energy, financial services, food and agriculture, government facilities, health care and public health, information technology, nuclear reactors, materials, and waste, transportation systems, and water and wastewater systems. (e) “Department” means the Department of Technology. (f) “Office” means the Office of Emergency Services. (g) “Operator” means a state agency responsible for operating, managing, overseeing, or controlling access to critical infrastructure. (h) “State agency” has the same meaning set forth in

Section 11000. 8954.51. (

a) On or before July 1, 2026, an oversight personnel for an operator that deploys a covered AI system shall establish a human oversight mechanism that ensures a human does both of the following:

(1) Monitors the artificial intelligence system’s operations in real time. (2) (

A) Except as provided in subparagraph (B), reviews and approves any plan or action proposed by an artificial intelligence system before execution. (

B) If oversight personnel determine that prior review and approval under subparagraph (

A) is substantially disruptive to the operation of the covered AI system, the operator shall instead implement a process for periodically reviewing the actions of the covered AI system to ensure accuracy and reliability. (b)

(1) The department shall develop specialized training in AI safety protocols and risk management techniques to be given annually to oversight personnel.

(2) An operator shall designate at least one employee to serve as oversight personnel who is responsible for administering the human oversight mechanism. The oversight personnel shall complete the annual training under paragraph (1). (c)

(1) Oversight personnel for an operator that deploys a covered AI system shall conduct an annual assessment of its covered AI systems that does all of the following: (

A) Evaluates the operator’s compliance with this section. (

B) Evaluates covered AI system performance and safety. (

C) Identifies and evaluates potential risks and vulnerabilities associated with the operation of the covered AI system, including those that could lead to mass casualty events or property damage in excess of five hundred thousand dollars ($500,000). (

D) Identifies any necessary updates to the human oversight mechanism used by the operator.

(2) Oversight personnel for an operator that deploys a covered AI system shall submit a

summary of the assessment findings to the department. 8954.52 The office shall not disclose any record or information within a record of the office related to this

article that is privileged, protected by copyright, or otherwise prohibited by law from being disclosed that is exempt from disclosure to the public under express provisions of the California Public Records Act (Division 10 (commencing with

Section 7920.000) of Title 1) or in which, based on the facts of the particular case, the public interest served by not disclosing the record clearly outweighs the public interest served by disclosure of the record.

SEC. 3. The Legislature finds and declares that

Section of this act, which adds

Section 8954.52 to the Government Code, imposes a limitation on the public’s right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of

Section of

Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest: To protect the sensitive information related to operating, managing, overseeing, or controlling access to critical infrastructure, it is necessary to limit the public’s right of access to these records.