Web sources

Have a Knowledge Base collect pages from a website and keep them up to date.

For: Owners and editors of a Knowledge Base · Last updated

The Web sources card is in the right column of a Knowledge Base page. It lists sites the Knowledge Base collects page content from. Each page it collects becomes a document named after the page's address. Everyone who can see the Knowledge Base can see its sources. Owners and editors can add, change, sync and remove them.

Add a source#

  1. Select Add a source.
  2. Enter the address in the URL field, for example https://docs.example.com/guides.
  3. Choose a Scope:
    • Single page
    • Pages under this path
    • Entire domain, which also shows a Depth field (it starts at 2)
  4. Choose how often it should update under Update: On demand, Daily, Weekly or Monthly.
  5. Select Add source. A message confirms the address was added and that its first sync is running now.

Addresses that cannot be collected, such as ones that lead to private network addresses, are refused with a message that says why.

Files linked from a page#

Besides pages, a source collects the files it finds linked from them: PDF, Word (.docx), Excel (.xlsx), PowerPoint (.pptx), OpenDocument (.odt, .ods, .odp), CSV, TSV, text and Markdown files, and images (.png, .jpg, .jpeg, .webp, .gif). Each file becomes its own document named after its address, read the same way as an uploaded file (see Adding and editing documents). A file that has not changed since the last sync is not read again, and a file that changed replaces the earlier text while that earlier text stays in use until the new one is ready. Open a file's document and select Open source to see the original.

A linked file of a supported kind that cannot be read leaves the rest of the source collected and makes the source show Partially synced. That covers older Office files (.doc, .xls, .ppt), files larger than the size your administrator allows, and files that are damaged or whose contents are not the kind of file the link says. The message names the file and the reason: one file reads "address was not added: reason", and several read "N linked files were not added. For example, address: reason". Linked files of other types, such as archives, are not collected and do not change the source's status. Partially synced also appears when a sync stopped at its page limit.

If a collected file later fails to read, its document shows Failed; select Try again on it (see Adding and editing documents).

The sources table#

Column What it shows
Source The address, its scope and, for a domain, its depth
Status Active, Syncing now…, Couldn't be crawled, Sync failed or Partially synced, with the reason when there is one
Last synced When it last synced, or that it has not synced
Update How often it updates. Change it from the list.

While a source is syncing, the table refreshes by itself.

Sync and remove#

  • Select Sync now to collect the pages again straight away. It shows Syncing… while it runs. A source that is already syncing cannot be started again.
  • Sources with a Daily, Weekly or Monthly update sync again on their own once a day, a week or a month has passed since their last sync. A source set to On demand syncs only when you select Sync now.
  • Select Delete and then Remove source to stop future syncs. The dialog notes that content already collected stays until you delete those documents.

Each collected page keeps its address, title and last-updated date, so the search filters in Browsing and searching can narrow results by site section or date.

When there are no sources the card says no web sources have been added.

To collect documents from a SharePoint site instead, see SharePoint sources.