Skip to content
SmiKar Software

Using the Forage Records Section

16 min read · Last updated · Page version 32

Forage appears as a Records section inside your Squirrel portal, alongside Archive and Security.

The Records navigation inside the Squirrel portal: Records holding Dashboard, Search, Coverage, Sites, Exceptions, Readers, Queue, Activity, Log, Reports and Settings, and Classification holding Overview, Search by Label, Labels and Content Rules

The menu has two groups.

  • Records: Dashboard, Search, Coverage, Sites, Exceptions, Readers, Queue, Activity, Log, Reports and Settings.
  • Classification: Overview, Search by Label, Labels, Suggested Labels and Content Rules. Where content classification is not switched on for a deployment, this group shows a padlock.

Dashboard

The Records dashboard: a blue progress panel naming, in turn, the site being read and the site being checked for changes, the files already searchable, the rate an hour and an estimate of the days left, over a bar splitting every listed file into searchable, being read, waiting and could not be read; then tiles for files listed, searchable, waiting and could not be read, and for the sites and libraries included; then progress in SharePoint and in the archive side by side, a ring of how much of each file was read, files and data made searchable per day and files per hour, the changes found in the libraries day by day, the readers that are running, the latest problems and a summary of what can be found in Search

The overall picture:

  • which site is being read now, and how far through the estate Forage is;
  • how many documents Forage is tracking, and how many are searchable, waiting, and could not be read;
  • document libraries read and still to read;
  • progress by source, live and archived;
  • documents made searchable per day and per hour, and an estimate of the time to finish at the rate actually achieved.

The estate total is only as current as Squirrel's last scan of each site, which the page states. Other figures refresh on a short cycle.

How to: follow Forage's progress.

One full-text search across live and archived content. Results carry highlighted extracts, the file's name and location, and the retention and sensitivity labels Squirrel recorded for it. A file still in SharePoint links straight to it; a file in the archive links to the Archived Files page instead, ready to restore.

Search tips, as the page gives them:

  • Put a phrase in quotes to find those words together and in order.
  • Use | between words for OR.
  • Put - in front of a word to exclude it.

Filter results by All, Live or Archived, and open any result to read its text.

Forage search results for a phrase, each hit showing the file name, its full path in SharePoint, when it was modified and its size, the labels its content rules gave it, and extracts from the text with the matching words highlighted, with an open link on files still in SharePoint and a restore link on files in the archive; one result was found by its file name alone and says it was not read because Forage cannot read that type of file

Two things worth knowing before you rely on it. Search covers only what Forage has read so far, which the Coverage page quantifies. And results are not limited to what each person could open in SharePoint.

How to: search, and see why a file was not read.

Coverage

Coverage keeps three answers apart: how much of the files in scope is searchable, how much of all of SharePoint is, and how much of what Forage has read so far is. It also shows whether what was read is being kept current.

It shows:

  • how much of the files in scope is searchable, how many are still to read, how many have a problem now, and how many sites are listed;
  • where every file in scope stands: searchable, waiting or being read, could not be read, listed but not read, or not listed yet;
  • the reading rate, the files found new in the last hour, when sites were last checked for changes, and the first reads under way;
  • the sites Forage has started but not finished, with the type of read, files listed and time so far;
  • under tabs, coverage by file type, the problems now, what was listed but not read and why, how files were read, and all of SharePoint for reference.

Everything Forage does not read is still counted, with the reason. It is accounted for, not missing.

The Coverage page: four figures for the files in scope that are searchable, the files still to read, the problems now and the sites listed; a bar showing where every file in scope stands; tiles for the reading rate, the files found new in the last hour, the last change check and the first reads under way; the sites in progress with the type of read, files listed and time so far; then tabs for coverage by file type, problems, files not read, how files were read and all of SharePoint, open on the table by file type

Sites

Every site on Squirrel's list, read or not, with its file count and size, whether Forage reads it, its rank, and how far reading has got: not started, reading, done, excluded or not decided. This is also where you choose what Forage reads.

  • Include or exclude sites, one at a time, in bulk, or by an address pattern.
  • Rank them, so the sites that matter are read first. Rank 1 is read first, 2 second; an unranked site is read after every ranked one.
  • Choose what happens to new sites, and to a site you exclude after Forage has already read it: keep what was read searchable, or remove it from search.

A new deployment reads nothing until sites are included. Forage follows each change from its next pass over the sites, every change is recorded with who made it and when, and excluding a site never touches the site itself.

How to: choose which sites Forage reads.

The Sites page: counts of the sites included, excluded and not yet decided and the files in the included sites, then one row per SharePoint site showing whether Forage reads it, how many files and how much data it holds, its rank, how far Forage has got with it and whether it has finished, when Forage last checked it for changes and when someone last edited it, with buttons to edit a site, visit it, or include and exclude sites in bulk

Exceptions

What could not be read, grouped by cause and ranked by documents lost.

The Exceptions page: the files that could not be read split by whether the cause is the document itself or SharePoint's copy, or something else, then every reason ranked by how many files it stopped, with how many sites it affected, how often it happened in the last 24 hours and what Forage does next about it; underneath, the sites with the most unread files and their main reason, an explanation of each kind of problem, and a list of every individual problem newest first

  • which causes are still happening;
  • which kind of cause each is, and so who can act on it;
  • whether each cause will be retried or needs someone to act;
  • which sites lost documents;
  • the documents behind each cause, with filters, and an export of the current page of that list.

Failures that belong to the document itself, such as a password or encryption, or to a copy SharePoint held that was already damaged before Forage fetched it, are labelled as such so nobody chases them as faults.

How to: understand exceptions and unread files.

Scanned pages that could not be read are not yet listed here. Their documents are counted as partly read.

Activity

The Activity page: how many files were read in the last hour and the last 24 hours, then a feed of entries with the time, the kind, who did it, what happened, what it was done to and the detail, including searches, a settings change showing its old and new value with a warning that it has not been applied to existing files, and files that could not be read with the step and the reason

  • what has been read;
  • what an administrator changed;
  • who searched, and for what;
  • what would not open;
  • documents Squirrel reported as changed where the content turned out to be identical, so nothing needed reading again.

The two processing status pages

The Records section also has processing status pages for administrators.

Log

One timeline of what people did, newest first. The Log records every change to settings, content rules, labels and sites, every search, every document opened and every report exported, each with who did it, when, what it was done to and the reason given.

Open an entry to see the value before and after the change, and the detail recorded with it. Filter by kind, by person, by any text, and by date, then export to PDF or a spreadsheet.

The Log is tamper-proof: no one using Forage, administrators included, can change or remove an entry, and clearing Forage's data to read the estate again keeps them. Sign-ins to the portal are not recorded.

The Log: filter buttons for settings, rules, labels, sites, data resets, files opened, reports exported, searches and other, with boxes for who, for any text and for a date range; then the entries newest first, each showing when, its type, who did it, what they did, what it was done to and the reason they gave, including settings changes, a rule activated, a label edited, searches, an exported report, an emailed report and a file opened in the portal

How to: see who did what.

Reports

Four reports, each filtered by site and by date and exported as a PDF or a spreadsheet:

  • All your files: files by site, by SharePoint or Archive, and by type, with how much was read in full, read in part, or not read, and why.
  • Sensitive content, where content classification is enabled: the files carrying each label, by site and severity.
  • Labels by file, where content classification is enabled: one row per file, with the labels on it and when it was last checked.
  • Files not read: what was not read, and the reason.

Every export states the customer, when it was produced, and the filters used. Lists show up to 100 rows; the summary figures count everything the filters select.

The Reports page with "All your files" chosen: buttons for the four reports, filters for site and for the dates a file was last changed, then the totals for files, read in full, partly read, waiting for OCR, waiting to be read and not read; then the same figures broken down by where the files are kept, by site and by file type, and last a table of why files were not read

How to: produce a report for a board or an auditor. Forage can also email the reports on a schedule.

Classification Overview

Where content classification is enabled, one page showing how far checking has got and what has been found.

  • how many files have been checked out of all the files that can be checked, the share done, and whether checking is running;
  • files checked, waiting, labelled, the active labels and rules, and files that could not be checked;
  • how many files carry each label, and the sites holding the most labelled files;
  • the files most recently labelled, and the most recent changes to rules.

A file is checked once Forage has read it, and Forage reads a file's pictures and scanned pages in the same pass as its typed text, so a file is checked on all of its text. Files with no text whose scanned pages were never read are shown apart: they have nothing to check unless they are read again. Checked means compared against the rules active now. The totals are prepared in the background every few minutes, and each says how old it is.

The Classification overview: a blue panel giving the share of files with text checked, the files checked, waiting and labelled, and a bar splitting them into checked, never checked and checked under earlier rules; then tiles for files checked, waiting, labelled, the active labels and rules and the files that could not be checked; then what each rule checker is doing and what is waiting to be checked with an estimate of the time to finish; then a chart of how many files carry each label and a list of the sites with the most labelled files; and last the files labelled most recently and the most recent changes to rules

How to: see classification at a glance. You can also find files by label in Search.

A search by label in Forage: the label filter open under the search box with a chip for each label, two of them selected, a severity list and a choice between the selected labels and any label, a note that Forage is still adding labels to Search so a search by label may miss some files, and then the files carrying either label, each showing its name, its path and the labels it has

Labels

Where content classification is enabled:

  • how much of what Forage has read has been checked against the current rules;
  • each label with its document count and severity;
  • create, edit, retire, reinstate and delete your own labels, with a history of every change;
  • the rules behind the labels, with the documents behind each.

The Labels page: the number of active and retired labels, how many labels have been given and how many rules give them, then how much of what Forage has read has been checked against the rules that are active now, then a card for each label with its category, its severity, what it means and how many files carry it, and underneath a table of the rules that give labels with each rule's version and how many files it has found

Counts are within the documents checked, not the whole estate, and are counted in documents rather than findings. When the rules change, the checked figure falls and refills as Forage checks everything again in the background.

New to this? Start with labels and rules in plain words.

How to: create and edit labels, and see what a label or rule found.

Suggested Labels

Where content classification is enabled, the labels Forage suggests from the files it has read.

  • choose the sites to look through, and follow the steps while suggestions are prepared;
  • each suggestion with its file count, example files, keywords and sites, and how many files a rule on those keywords would find;
  • edit a suggestion's name, category, severity and words before deciding, then accept it, which creates the label and a draft rule, use it for an existing label, merge it into another suggestion, or reject it.

Forage AI, which runs inside your own deployment, reads the files Forage has read there, groups them by what they contain and suggests a label for each kind of record it finds, such as invoices, packing lists or test reports, with how many files in those sites such a rule would find. Nothing is labelled until a person accepts a suggestion and switches its rule on, a rejected suggestion is not made again, and every decision is recorded in the Log. Reading slows down while suggestions are prepared, and returns to normal speed by itself when they are ready.

The Suggested Labels page: tiles for the suggestions to review, the files covered out of those read in the chosen sites, the suggestions accepted and the kinds of record found; then a card for each suggestion with its name, what it covers, its severity and category, how many files it covers and in how many groups, how many files a rule on its keywords would find and how many of a sample fit, the keywords a rule would look for, the sites they are in, a link to example files, and buttons to accept it, use it for an existing label, merge it or reject it

How to: get label suggestions.

Content Rules

Where content classification is enabled, this is where a rule is written, saved as a new version, test run, activated, retired, or deleted while still an unused draft.

Every change is a new version. Rules are not edited in place.

The Content Rules page: a card for each rule showing the label it gives, whether it is active, which version is live and how many files it has labelled; underneath, the selected rule written out in plain words with what each of its detectors looks for, its pattern, its number check and the words that must appear nearby, and a table of every version of that rule with who created it and when

A test run before a rule goes live shows three figures:

  • the most the rule could match;
  • a sample checked with the real rule;
  • a projected range across the documents read so far.

The test run is offered rather than required, and its projected range covers the documents read so far rather than the whole estate.

How to: write, test and switch on a content rule. There is also a worked example, from a new label to its results.

Settings

The Forage Settings page: where to read from and what happens to files of a type that is switched off, a summary banner, option cards for where to read from and what happens to a type that is switched off, a card for each family of file types with each type switched on or off, the file types Forage cannot read yet with how many files of each, the readers Forage has running, and the emailed reports section with who receives them, which reports, how often and from which sender address

  • Read live content, archived content, or both.
  • Choose which document types are read.
  • Choose what happens to documents of a type that is switched off: Keep them, or Delete them permanently.

Delete them permanently requires a reason, which is recorded. It removes Forage's copy of the text, its search entries, and its record that those documents were examined. The documents themselves, in SharePoint and in the archive, are untouched.

Every change is recorded with who made it and when, and takes effect automatically.

Which document types a deployment can read is confirmed by SmiKar. A new deployment starts with the most common types.

Questions go to sales@smikar.com.

Previous
← How Forage Works
Next
Why Archived Content Needs Its Own Records Tool →

More in Squirrel

See all pages →
Why Archived Content Needs Its Own Records Tool
A Worked Forage Example: From a New Label to Its Results
Choose Which Sites Forage Reads
Create and Edit Forage Labels
Find Files by Label in Forage Search
Follow Forage's Progress