Add Sources

This time, we will proceed to check the operation of the contents of www.sitecore.com. When actually working on the site, please read the domain part differently and understand the data structure of the site as you proceed.

First, create a new source.

Then open Web Crawler Settings and specify the domain.

The site sitecore.com is excluded from crawling below /search because of the way the search engine works. Set Glob Express for Exclusion patterns and /search for the value.

Also, as described in the previous article on obtaining sitemap.xml, set the User Agent and then configure it to crawl.

As for Available Locales, we will immediately assume en-us only.

Triggers is proceeding on the assumption that sitemap.xml is used, so we will use sitemap.xml this time.

For the Document Extractor, this time we will set up JavaScript.

In the source code, we will try to run the program with just the default code for the first time.

// Sample extractor function. Change the function to suit your individual needs
function extract(request, response) {
    $ = response.body;

    return [{
        'description': $('meta[name="description"]').attr('content') || $('meta[property="og:description"]').attr('content') || $('p').text(),
        'name': $('meta[name="searchtitle"]').attr('content') || $('title').text(),
        'type': $('meta[property="og:type"]').attr('content') || 'website_content',
        'url': $('meta[property="og:url"]').attr('content')
    }];
}

This completes the initial setup. Once you have done this, please publish your site and check that it crawls properly. After a few moments, 1000 contents have been added.

Document Extractor changes

In terms of retrieving and storing data from the HTML structure, the Javascript code described above works much the same as XPath. Therefore, this time, we have rewritten the type as follows so that the setting can be changed using URLs.

function extract(request, response) {
    $ = response.body;

    let url = request.url;
    let subtype;

    if (url.includes('/products/')) {
        subtype = 'Products';
    } else if (url.includes('/solutions/')) {
        subtype = 'Solutions';
    } else if (url.includes('/knowledge-center/')) {
        subtype = 'Knowledge Center';
    } else if (url.includes('/partners/')) {
        subtype = 'Partners';
    } else if (url.includes('/company/')) {
        subtype = 'Company';
    } else {
        subtype = 'website';
    }

　　　　return [{
　　　　　　　　'title': $('meta[name="searchtitle"]').attr('content') || $('title').text(),
　　　　　　　　'subtitle': $('meta[name="description"]').attr('content') || $('meta[property="og:description"]').attr('content') || $('p').text(),
　　　　　　　　'description': $('meta[name="description"]').attr('content') || $('meta[property="og:description"]').attr('content') || $('p').text(),
　　　　　　　　'name': $('meta[name="searchtitle"]').attr('content') || $('title').text(),
　　　　　　　　'type': subtype ,
　　　　　　　　'url': $('meta[property="og:url"]').attr('content')
　　　　}];
}

Change the settings and crawl again. After a while, the data is aligned in the content as follows.

When content types are added to the filter, candidates are displayed as shown below.

Summary

In this case, we used the URL to specify the content type, and while it is effective to include the data contained in the og tag, there may be cases where older content does not contain an og tag.

Tags

• Search