Forage Limitations

This page states plainly what Forage does not do, how today's features behave at their edges, and what is designed but not yet available.

Part 1: scope, by design

Sources. Forage covers SharePoint Online and the content Squirrel has archived from it. OneDrive, Exchange mailboxes, Microsoft Teams messages and other systems are not in scope.

Squirrel is required. Forage builds its picture of the estate from Squirrel's record, so a site or file that Squirrel does not record is not seen by Forage, and Forage is only as current as that record. Forage reads the sites on Squirrel's list of sites. Files Squirrel holds for a site that is not on that list are outside the estate and are not read, and the Coverage page shows how many there are.

Documents only. Forage reads documents. Video and audio are not read.

The current version only. Forage reads the current version of each document. Earlier versions, whether in SharePoint's version history or kept inside an archived file, are not read.

Nothing is left out silently. Files Forage does not read are still counted, with the reason: a type it does not read, a type an administrator has switched off, or a file that could not be read. The one exception is a switched-off type whose records an administrator has chosen to delete.

Part 2: how today's features behave

Reading documents

Protected files. Forage never removes or bypasses a password or encryption. A password-protected or encrypted PDF or modern Office file is recorded as protected and is not read; this includes Office files encrypted by a sensitivity label or by rights management. An older-format Office file with a password may instead be recorded as unreadable or partly read; its content is not searchable either way. A PDF that opens without a password but restricts printing or copying is read normally.

Formats read in part. Older Word and PowerPoint files are read for their text but not their layout, and are marked as partly read. Attachments inside email messages are listed by name; their contents are not read as part of the message.

Formats not read today. Some formats are not read today, including binary Excel workbooks, compressed (zip) folders, and design files other than drawings (.dwg, saved in any version of the format from 1984 on) and exchange drawings (.dxf). They are counted, with the reason. A drawing (.dwg) saved in one of the first versions of the format, from 1982 and 1983, is listed and findable by its name and folder, shown as "Not read yet: drawing version", and counted apart from the files that could not be read.

Pictures inside documents. Words in pictures are read with optical character recognition: in pictures embedded in Word, Excel and PowerPoint files, older formats included, and in pictures on PDF pages that also carry typed text. They are read in the same pass as the rest of the document, so a document's typed text and the words in its pictures become searchable together. A picture on a PDF page whose own text already covers it, such as a scan that was already made searchable, is left to that text. A picture in a format Forage cannot open is not read, and the document is marked as partly read. Words read from a picture are read the way any scan is read, so the cautions below about handwriting and poor scans apply to them too.

Very long documents. Forage reads a long document to its end: there is no limit on the amount of text or the number of pages it reads. Time limits apply to single steps, not to whole documents. A scanned page that cannot be read in time is tried again with more time, and if it still cannot be read, the document is marked as partly read. A PDF that stalls on a page, or an older-format Office file that takes too long to read, is recorded as not read, with the reason.

Scanned pages

Pictures with no text. Photographs, drawings and other pictures with no text have nothing to search. Forage does not describe what a picture shows, and describing pictures is not planned.

Handwriting and poor scans. Handwriting is read poorly or not at all. Faint or skewed pages are often read poorly. A page read with low confidence is still made searchable, and the record of how the document was read marks it as a low-confidence reading; any page read this way can contain misread words.

Scripts that have not been tested. Scanned pages are read in Latin script, accented letters included, and in Cyrillic script. Scanned pages in other scripts, such as Chinese, Japanese or Arabic, have not been tested, and Forage makes no claim about how well they are read.

Pages that mix scripts. A page that mixes scripts, such as Latin and Cyrillic text on one page, is read poorly: part of it is lost.

Scripts and orientation. Forage recognises the script and orientation of each scanned page and reads it accordingly, so it also reads accented Latin-script text, Cyrillic-script text and rotated pages. Documents already read are not read again automatically.

Search

Search languages. Search matches forms of the same English word. In other languages it matches the word as written, ignoring accents. Text in Chinese or Japanese characters is not searchable today.

Results are not trimmed to permissions. Search results are not limited to what each person may open in SharePoint.

What has been read so far. Search and findings cover only what Forage has read so far. The Coverage page shows how much of the estate that is, and its estate figures are a periodic count that says when it was taken, not a live figure.

Content rules and labels

Rules match exactly. Content rules match exactly the words and patterns written into them, ignoring capital letters, with checksum checks and nearby-word conditions where a rule asks for them. They make no judgement about meaning. Unlike search, a rule does not match other forms of a word: each form it should find has to be listed.

Identifier coverage and accuracy. Built-in checksum checks cover payment card numbers, international bank account numbers, Australian business and tax file numbers, and UK NHS numbers. Identifiers from other countries need rules written for them, without a built-in checksum. No accuracy figure is measured for any rule; a test run's checked sample is the evidence.

Findings cover what has been checked. Findings cover only documents Forage has read and checked against the rules active now. When the rules change, Forage checks everything again everything in the background and the counts refill. Forage keeps the current finding for each document and rule; when a new version of a rule examines a document, the earlier version's finding is replaced, not kept as history.

Suggested labels. Suggested labels come from groups of files whose text is alike, among the files Forage has read in the sites an administrator chooses; files that do not belong to a large enough group get no suggestion. A suggestion's name, category, severity and keywords are a starting point for a person to check and edit. While suggestions are being prepared, reading slows down, and it returns to normal speed by itself when they are ready; one set of suggestions is prepared at a time.

Microsoft labels. Search results show the retention label and the sensitivity label that Squirrel recorded for each file. Forage never applies, changes or removes a label, does not act on retention labels, and does not use sensitivity labels in rules or as a search filter.

Keeping current and reporting

Changes arrive on a cycle. Forage picks up changes on a regular sweep, not instantly, and can be no more current than Squirrel's own record, which Squirrel updates on its own schedule. The sweep keeps up to date the sites Forage has already read; sites it has not read yet, including new ones, are read as part of reading the estate.

Files moved between sites. A file moved into a site without being changed may not be noticed by the regular sweep. A full re-check of the estate finds it, and is run deliberately rather than as part of the sweep.

Scanned-page failures. Scanned pages that could not be read are not yet listed on the Exceptions page; their documents are counted as partly read.

Archived content, access and audit

Azure's offline archive tier. Where Squirrel has placed an archived file in Azure's offline archive tier, Forage asks Azure to bring it back online before reading it. Azure charges for this, it can take up to a day, and Forage does not return the file to the offline tier afterwards.

Access is all or nothing. There are no narrower roles today. Every Full Administrator who can use the Records section can use every page in it: search across all content, the rules, and the settings.

What the Log records. The Log records changes to settings, content rules, labels and sites, data resets, searches (the words used and who searched), documents opened in the portal, report exports and emailed reports, whether sent or not, and every test email asked for, each with who did it, when, why, and the before and after where that applies. It does not record search results, by design, because that would copy sensitive content into the Log.

The Log is not a guaranteed-complete record. Documents opened and reports exported are not recorded while a supporting service is down, so the record of openings and exports is not guaranteed to be complete.

What tamper-proof means here. No one using Forage, administrators included, can change or remove an entry. That is a property of the product, not of the storage underneath it, so it is not write-once storage and Forage does not claim to be.

Sign-ins are not recorded. Who signs in to the portal is not yet recorded.

An audit export packaged for regulators. The Log exports as CSV or PDF, but an export packaged for regulators with an integrity check is not yet available.

Part 3: not yet available

• Roles narrower than Full Administrator, such as records manager, reviewer and search operator.
• Limiting a search to a case.
• Exporting search results, including for eDiscovery.
• A report of documents holding sensitive content that do not carry a matching sensitivity label.
• Suggested terms for a rule, and a measured precision for each rule.
• Recognising identity documents in images.

Questions about how a deployment is secured go to SmiKar (sales@smikar.com).