Connect custom website to Rovo

This connector allows you to a do a search crawl and index of your website to show in Rovo Search results and use in Rovo Chat and Agents.

Before you begin

• This connector can only crawl publicly available websites that you own.

• To ensure you own the domain or subdomain, you need to be able to edit the robots.txt file in the website you’d like to crawl.

• We encourage you to review which pages are publicly available on your website. Pages that have not been restricted (and found by the crawler) will appear for your teammates in search results. This connector currently only crawls publicly available websites.

Editing your robots.txt

You need to be able to edit the robots.txt file on your website. If you’re not sure what a robots.txt file is, see How to write a robots.txt file, or talk to your website administrator.

At minimum you’ll need to add the following to your existing robots.txt file on your website:

Note that User-agent: * will not permit crawling alone, the atlassian-bot line must be included.

You can additionally add specific allow or disallow rules to the robots.txt and the connector will follow these rules, for example:

This rule would allow Rovo to index every public page on your site except the content under /not-useful/.

Connecting and crawling your website

To get to the setup screen for your custom website in Atlassian Admin:

1. Go to admin.atlassian.com. Select your organization if you have more than one.

2. Select Settings > Rovo.

3. Under the Sites tab, next to the site you want to connect, select the More actions (•••) and select Add connector.

4. Select Custom website and press Next.

To setup your crawl:

1. Enter a website name for the site you’d like to crawl.

2. Add the full URL of the domain. Include the protocol (https://).

3. Choose how often Rovo should index your site.

4. Review and agree to the data usage information.

5. Select Connect.

Next steps

After you’ve finished setting up the crawl:

1. The crawling and indexing of your site will start immediately.

2. Pages will start to show in Search incrementally for you and your team over the next few hours.

3. Depending on the number of pages on your website, it may take some time for all your website’s content to be indexed and appear in Search.