Release notes: Aug 13, 2025 (v8.272.0)

Inference Red-Team

More options for rate-limiting: Max concurrent requests

Many organizations are using smaller models with lower throughput for bespoke use cases. The Max concurrent requests setting allows them to attack and test those models without encountering rate limiting errors.

The Max concurrent requests setting caps the number of concurrent attacks, a more nuanced way of rate-limiting than requests per second (RPS). A low RPS can still overwhelm a small model by sending too many attacks concurrently.

• For low-throughput models that can handle concurrency, use Max concurrent requests or Max requests per seconds and set a low integer value.

• For models that can only handle sequential attacks, use Max concurrent requests and set the value to 1. In this configuration, the system will sends one request, wait for it to finish, then send the next.

To access this setting, navigate to Connections, select a provider, then select locate model you want to rate limit.  In the three dots menu, select Manage model then Advanced configuration.

Inference Defend

Audit, block, and now… redact!

Redaction is one of our most asked-for features and we’re excited to put this capability into customers' hands. Now you can configure regex and keyword scanners to block, audit, or redact – giving you more options for controlling how sensitive data is transmitted to and from models.

While blocking unauthorized or problematic prompts and responses gives maximum security, it also makes for a less-than-optimal end-user experience. Instead of blocking the whole prompt or response, we can now redact just the content matching the scanner definition. Redactions means:

• Users can still get a response even if part of the content is redacted.

• The feedback of seeing the redacted content helps train users to understand what’s safe and permitted and what isn’t.

As part of this enhancement, you’ll see the following changes in the UI:

• Matching content is replaced with asterisks (*****).

• In the UI, the Block/Audit toggle has changed to a selector that includes Redact as an option for keyword and regex scanners.

• Scanners set to redact also appear in a separate section in the Outcome analysis tool.

• This control was previous labeled Status; it’s now labeled Mode.

Key capabilities

• Works with both regex and keyword custom scanners.

• Replaces matched content with asterisks in both user prompts and model responses.

• Redacted content is not stored in prompt history logs, making it easier to comply with data privacy rules.

• Applies consistently to all users, including org admins.

• Redaction runs before other scanners like prompt injection or PII, so downstream tools only receive the safe, redacted version.

• In Playground sessions, tests will display as Blocked with the name of the scanner created for redaction.

Bug fixes

• The Playground background color was incorrect, making the side panel invisible. Resolution: Fixed.

• The UI was not handling the new failure_limit_reached error type from the campaign-runs/{id}/summary endpoint. Resolution: The UI now has specific handling for this error type. It is not displayed in the side panel to avoid redundancy, as it's caused by other errors.

• The prompt history page would break when opening certain “Prompt and responses” tabs. Resolution: Fixed.

• An unexpected error occured on the Audit logs page. Resolution: Fixed.

• An error message was not appearing when a user started the email field on the "invite user" tab with a space. Resolution: Fixed.

• The table was refreshing after a user canceled a report run. Resolution: Fixed.

• One of the sort dropdown menus was too wide and misaligned. Resolution: Fixed.

• When selecting a scanner, the text would overlap with the filter text. Resolution: Fixed.

• After selecting a configuration during application creation, the key and value fields were disappearing. Resolution: Fixed.

• The "save" button in the scanners version modal was not being disabled while a request was in progress. Resolution: Fixed.

• The "API tokens" table on the project detail page had incorrect capitalization for column headers. Resolution: The headers "ID Name" and "Expires At" have been changed to "ID name" and "Expires at."

• The "view" buttons on project cards were not aligned. Resolution: Fixed.

• The three-dot menu was visible on projects a user had created, even though the only available action was to leave the project, which was not possible. Resolution: Fixed.

• The dropdown in the pagination was not disabled when there were fewer than 10 items to display. Resolution: Fixed.

• The campaign name was not consistently displaying on the report panel, especially after reloading the view or navigating back to the report. Resolution: Fixed.

• The font for "no reports" was incorrect. Resolution: Fixed.

• Error logs for custom connections were not showing in the UI, even when the user was directed to check them. Resolution: Fixed.

Known issues

• Campaign Runs: The /campaign-runs endpoint is not correctly filtering by the status parameter for in_progress and error states. It is returning runs with a complete status or fewer results than expected. This issue also appears to affect the cancelling and cancelled statuses.

• Reports: The reports table is not rendering correctly on narrower screens.

• Scanners: After enabling or disabling a group of scanners within a CAI scanner package, the checkboxes for individual scanners become unresponsive until another action is taken.