Protect OpenSearch
Snapshots, Marks, Proof, Rewind, Pulse, and users and roles for OpenSearch 2 and 3.
Rowsafe backs up OpenSearch with its own snapshots, sent to your bucket and encrypted on your server. You get Marks, a weekly Proof, Rewind (bring documents back or rewind the whole server) and Pulse with fixes. Some features aren't there yet: see Not yet.
Restores go back to a snapshot
OpenSearch keeps no change log, so Rowsafe can't restore to any second. You pick a snapshot or a Mark instead:
- A snapshot is taken every 5 minutes, copying only what's new. You lose at most 5 minutes. Owners can pick 15, 30 or 60 minutes in Settings.
- A Mark takes one on the spot. Make one before a migration, reindex or bulk delete.
If the snapshot you picked was removed, a copy uses the nearest older one; a rewind in place stops and names it.
Before you start
- OpenSearch 2 or 3, as one node. Clusters, and servers that answer like Elasticsearch, are refused.
- Debian or Ubuntu, with the security plugin on or off.
- REST on
127.0.0.1:9200by default, over HTTPS (or HTTP if TLS is off). - The
opensearchprogram on the server, for Proof and Rewind copies. Backups work without it. - A bucket and a passphrase. See Adopt an existing database.
Turn on backups
Run this on the server, then approve it in your browser:
curl -fsSL https://rowsafe.sh | sudo sh -s -- --accountA folder for snapshots. The installer adds /var/lib/rowsafe-opensearch/snapshots to path.repo in opensearch.yml, keeping the old file as opensearch.yml.rowsafe-backup. The agent joins the opensearch group to read snapshots, never to write them.
Rowsafe's own OpenSearch user. With the security plugin on, an administrator (all_access, like admin) signs in once. Rowsafe creates rowsafe with a random password kept on the server. The admin password is never saved.
One restart. OpenSearch reads path.repo only at start. The installer asks first (about a minute; apps reconnect). Say no and backups start after your next restart.
The plan. Review it and say yes. Rowsafe checks the bucket and passphrase, then takes the first snapshot.
- Without a terminal, use
--protect NAMEwithROWSAFE_OPENSEARCH_ADMIN_USERandROWSAFE_OPENSEARCH_ADMIN_PASSWORD(used once). It never restarts OpenSearch. - If the demo setup (demo users, demo certificates) is still there, the installer warns and goes on. Pulse shows it until it's gone.
What Rowsafe does
| How | |
|---|---|
| Backups | OpenSearch's snapshot of every index and data stream, every 5 minutes, and a full one daily at 01:00 UTC. Only new files are copied. If snapshots get slow, Rowsafe spaces them out to use at most about a quarter of the server's time. |
| Encryption | The agent copies new snapshot files to your bucket (or Rowsafe Storage), encrypted on your server. OpenSearch never sees the bucket. |
| Keeping | Every snapshot for a day, one an hour for a week, and the daily full ones (7 by default) with every Mark since the oldest. Unneeded files are deleted, in the bucket too. |
| Marks | A snapshot taken on the spot. |
| Proof | Weekly: restore the newest snapshot from your bucket into a temporary local OpenSearch, check every index is green with all its documents, then delete it. |
| Rewind | A copy, compare, bring documents back, or rewind the whole server with Undo for 7 days (below). |
| Pulse | Health, shards, disk, memory, slow-downs and security (below). |
| Restart | Restart, after you confirm (with root's permission). |
| Databases & users | Indices and the security plugin's users (below). |
Rewind
In the dashboard, for owners and admins.
- Restore a copy at a snapshot or Mark, as a temporary OpenSearch on the same server that only Rowsafe signs in to.
- Compare an index with production by
_id: deleted, changed and added documents. Large indexes: first million documents (the result says so). - Bring back documents into the indices you pick, with their
_id. Existing documents are never overwritten unless you include changed ones. A deleted index is made again with the copy's mappings.
Rewind the whole server
Rewind the whole database puts every index and data stream back to a snapshot or Mark:
- Rowsafe snapshots everything as it is now, kept 7 days for Undo rewind.
- It deletes and restores each index and data stream. Newer indices are removed (they're in the Undo snapshot).
Nothing restarts. Searches of an index fail for the moments it's restored, and writes made meanwhile are replaced. If it fails, the first snapshot is put back.
Pulse
Pulse reads OpenSearch about once a minute: health (green, yellow, red), shards, indices, disk against the watermarks, heap, long garbage collections, refused requests (circuit breakers, full queues), and the last snapshot.
Each fix confirms first and checks the problem is still there:
| Finding | Fix |
|---|---|
| Health is yellow: replicas have nowhere to go (one server can never hold a copy of its own data) | Set replicas to 0 on those indices. Instant; no data moves. |
| Writes are still refused after the disk filled up | Allow writes again: removes the read-only block, only once the disk has room. |
The security check covers: security plugin off (also checked from the internet), TLS off, demo users with published passwords, demo certificates, the demo admin certificate (CN=kirk) trusted or its key readable by others, and port 9300 open beyond the server. Fixes need opensearch.yml and a restart, so each shows steps under Do it yourself.
Users and roles
Databases & users uses the security plugin's users. A "database" is an index or data stream.
- Create an index (empty), optionally with an owner. Remove one: Rowsafe saves a Mark first.
- Add a user on the indices or patterns you list (
logs-*): read only, read and write (also documents, creating those indices, mappings) or owner (everything on them). - New password and Remove user. Passwords are made on your server and shown only to you.
Built-in users (admin, kibanaserver...), reserved and hidden users, administrators (all_access) and Rowsafe's own are listed but never changed. Without the security plugin there are no users.
What Rowsafe may do
Rowsafe's user (rowsafe) reads health and stats, and manages snapshots in its own folder. Only when someone asks, it creates, writes and deletes indices, changes replicas, removes the read-only block, and manages users (its role is in plugins.security.restapi.roles_enabled).
Root decides what Rowsafe may do on the server (permissions, sudo rowsafe-allow): restart allows Restart. Rowsafe doesn't install OpenSearch updates yet.
On servers Rowsafe creates
Rowsafe Cloud and Create a server for me can install OpenSearch 3:
- OpenSearch's own packages (kept on 3.x), one node, with its bundled Java.
- Security plugin on, no demo users or certificates. One
admin, password readable by root only. - HTTPS on
9200. Port9300stays on the server. - Half the server's memory (at most 31 GB), Performance Analyzer off. 4 GB of memory or more.
- No standby, clones or private connections from AWS.
Apps connect over HTTPS on 9200 with a user from Databases & users.
Restore without Rowsafe
With the ROWSAFE_REPO_* lines of /etc/rowsafe/agent.env in the environment, decrypt a snapshot into a folder:
set -a; . /etc/rowsafe/agent.env; set +a
# --stanza is the database's "Bucket folder" (on its Settings page in the dashboard).
rowsafe-agent opensearch download-backup --stanza search
rowsafe-agent opensearch download-backup --stanza search --label 20261009-013000F --to /srv/os-restoreIt's a normal snapshot repository. Make it readable by OpenSearch, add it to path.repo and restart. The agent prints the two requests that register and restore it.
Not yet
- Restores to any second, and Find the moment.
- Updates and upgrades from the dashboard, Tuning, logs and recommendations.
- Standby servers, clones, Move in, safe copies and migration previews.
- Connection pooling, a second copy, and Files.
Limits
- Restores go back to a snapshot (every 5 minutes by default) or a Mark, never in between.
- One node only. No clusters, no Elasticsearch.
- Not in Docker yet. No OpenSearch agent image: install the agent on the server.
- Disk: snapshots are kept on the server too (
/var/lib/rowsafe-opensearch/snapshots). - Users aren't in backups. Security plugin users and roles and other system indices stay out. Rewinds leave index templates and ingest pipelines as they are now.
- Temporary servers (Proof, copies) need about 700 MB of free memory. The agent refuses if there isn't enough.
- Compare looks at the first million documents of an index.
- Data streams only take new documents, so changed ones are never overwritten.
- Rewinding the whole server makes searches of each index fail briefly and replaces writes made meanwhile.
- Security fixes need
opensearch.ymland a restart: steps you do yourself. - The first restart: backups start only after OpenSearch restarts once with Rowsafe's folder in
path.repo.