Visions In Code

Hello happy people, I hope you are enjoying the summer 🙂

Habitat is great and it’s always a joy to work with it. Again a big applaud to the team behind Habitat and Helix concept, great job guys!!
If you are new to Habitat, check these links and be amazed 🙂
Sitecore Helix Documentation
Sitecore Habitat

How about we devote a day for Helix/Habitat – Happy Helix/Habitat day 😎

In this post I would like to share with you guys how you can make your Habitat/Helix solution run on Solr. There are a lot of great posts on how to set up Solr for Sitecore and even posts on how to setup Solr for Habitat:

The Sitecore 8.2 with SOLR 6.2 blogposts series
Configuring Solr search with Sitecore 8
Configuring Solr for use with Sitecore 8
Setting up SOLR in Habitat

I could not find any posts on how to make Solr run with a Habitat solution, so lets make one 🙂

Lets break down the work into following steps:
1. Install/setup Solr – Here we will be using Docker
2. Make changes in the Habitat solution in order to make it work with Solr (Habitat is using Lucene)
3. Disable Lucene config files and enable Solr config files – Automate it by using PowerShell

1. Install/setup Solr

We need to setup Solr, why not set it up using Docker. I got inspired by Mr Laub – Anders Laub, he has created a great repository on the github – Docker containers for Sitecore development.
Lets follow his instructions and setup the Solr.

Download and install Docker.
Docker v17.06.0 or later – Download here
Don’t forget to register and share your C folder.

Download/clone Mr Laub’s great repository
at the github- Docker containers for Sitecore development

Notice all the lovely cores/indexes(with config files) that has been created by Anders:

If you have issues or some crazy ideas, please contribute to Ander’s great repository 🙂

Open Powershell as admin in /containers/solr/bitnami-6.6-sitecore/
Run > docker-compose up (I’m using Windows Powershell)
Wait for SOLR to start
Now it’s time to see if it’s working, check out the url: http://localhost:8983/solr

Notice that each index has its own core, sitecore-master-index, sitecore-web-index etc.

Lets test the sitecore-web-index, by browsing to this url:

http://localhost:8983/solr/sitecore_web_index/select?indent=on&q=*:*&wt=json

{
  "responseHeader":{
    "status":0,
    "QTime":0,
    "params":{
      "q":"*:*",
      "indent":"on",
      "wt":"json"}},
  "response":{"numFound":0,"start":0,"docs":[]
  }}

Ok cool 🙂 So now we have a running Solr, next will be to connect it to our habitat solution and fill the indexes(cores) with data.

2. Make changes in the Habitat solution

Habitat is using Lucene so we need to do some changes in order to make it work with Solr.

Here are the projects that needs to be changed/updated:
Sitecore.Foundation.Indexing
Sitecore.Foundation.LocalDataSource
Sitecore.Feature.News
Sitecore.Feature.PageContent
Sitecore.Feature.Person

Ok, we will start by changing the config files that contains indexConfigurations.
Foundation.Indexing.config – Right now the config is for Lucene, lets change it for Solr. Replace the content in the node indexConfigurations with the following:

<defaultSolrIndexConfiguration>
	<fieldMap type="Sitecore.ContentSearch.SolrProvider.SolrFieldMap, Sitecore.ContentSearch.SolrProvider">
	  <fieldNames hint="raw:AddFieldByFieldName">
		<field fieldName="all_templates" returnType="stringCollection" fieldNameFormat="{0}_sm"  multiValued="true" storageType="YES" indexType="TOKENIZED" vectorType="NO" boost="1f" settingType="Sitecore.ContentSearch.SolrProvider.SolrSearchFieldConfiguration, Sitecore.ContentSearch.SolrProvider">
		  <Analyzer type="Sitecore.ContentSearch.LuceneProvider.Analyzers.LowerCaseKeywordAnalyzer, Sitecore.ContentSearch.LuceneProvider" />
		</field>
		<field fieldName="has_presentation" returnType="bool" storageType="YES" indexType="UNTOKENIZED" vectorType="NO" boost="1f"  settingType="Sitecore.ContentSearch.SolrProvider.SolrSearchFieldConfiguration, Sitecore.ContentSearch.SolrProvider" />
		<field fieldName="has_search_result_formatter" returnType="bool" storageType="YES" indexType="TOKENIZED" vectorType="NO" boost="1f"  settingType="Sitecore.ContentSearch.SolrProvider.SolrSearchFieldConfiguration, Sitecore.ContentSearch.SolrProvider" />
		<field fieldName="search_result_formatter" returnType="string" storageType="YES" indexType="UNTOKENIZED" vectorType="NO"  settingType="Sitecore.ContentSearch.SolrProvider.SolrSearchFieldConfiguration, Sitecore.ContentSearch.SolrProvider" />
	  </fieldNames>
	</fieldMap>
	<virtualFields type="Sitecore.ContentSearch.VirtualFieldProcessorMap, Sitecore.ContentSearch">
	  <processors hint="raw:AddFromConfiguration">
		<add fieldName="content_type" type="Sitecore.Foundation.Indexing.Infrastructure.Fields.SearchResultFormatterComputedField, Sitecore.Foundation.Indexing"/>
	  </processors>
	</virtualFields>
	<documentOptions type="Sitecore.ContentSearch.SolrProvider.SolrDocumentBuilderOptions, Sitecore.ContentSearch.SolrProvider">
	  <fields hint="raw:AddComputedIndexField">
		<field fieldName="has_presentation" storageType="YES" indexType="untokenized" >Sitecore.Foundation.Indexing.Infrastructure.Fields.HasPresentationComputedField, Sitecore.Foundation.Indexing</field>
		<field fieldName="all_templates" storageType="YES" fieldNameFormat="{0}_sm"  multiValued="true" indexType="tokenized" >Sitecore.Foundation.Indexing.Infrastructure.Fields.AllTemplatesComputedField, Sitecore.Foundation.Indexing</field>
		<field fieldName="has_search_result_formatter" storageType="YES" indexType="untokenized" >Sitecore.Foundation.Indexing.Infrastructure.Fields.HasSearchResultFormatterComputedField, Sitecore.Foundation.Indexing</field>
		<field fieldName="search_result_formatter" storageType="YES" indexType="untokenized" >Sitecore.Foundation.Indexing.Infrastructure.Fields.SearchResultFormatterComputedField, Sitecore.Foundation.Indexing</field>
	  </fields>
	</documentOptions>
</defaultSolrIndexConfiguration>

Foundation.LocalDatasource.config – Right now the config is for Lucene, lets change it for Solr. Replace the content in the node indexConfigurations with the following:

<defaultSolrIndexConfiguration >
  <fieldMap type="Sitecore.ContentSearch.SolrProvider.SolrFieldMap, Sitecore.ContentSearch.SolrProvider">
    <fieldNames hint="raw:AddFieldByFieldName">
      <field fieldName="local_datasource_content" returnType="string" storageType="YES" indexType="TOKENIZED" vectorType="NO" boost="1f" settingType="Sitecore.ContentSearch.SolrProvider.SolrSearchFieldConfiguration, Sitecore.ContentSearch.SolrProvider">
        <Analyzer type="Sitecore.ContentSearch.LuceneProvider.Analyzers.LowerCaseKeywordAnalyzer, Sitecore.ContentSearch.LuceneProvider" />
      </field>
    </fieldNames>
  </fieldMap>
  <documentOptions type="Sitecore.ContentSearch.SolrProvider.SolrDocumentBuilderOptions, Sitecore.ContentSearch.SolrProvider">
    <fields hint="raw:AddComputedIndexField">
      <field fieldName="local_datasource_content" storageType="YES" indexType="TOKENIZED">Sitecore.Foundation.LocalDatasource.Infrastructure.Indexing.LocalDatasourceContentField, Sitecore.Foundation.LocalDatasource</field>
    </fields>
  </documentOptions>
</defaultSolrIndexConfiguration>
</indexConfigurations>

Ok that was easy, now we need to identify the indexed fields. The reason is that in Solr the indexed fields will look like something like this: title_t, summary_t etc (In Lucene it’s just title, summary).

If you have studied the Habitat code you will see that it’s using predicate builder for the search. In the Search method in Sitecore.Foundation.Indexing.Services.SearchService you will see it’s calling method AddContentPredicates(queryable, query)

public virtual ISearchResults Search(IQuery query)
{
  using (var context = this.SearchIndexResolver.GetIndex(this.ContextItem).CreateSearchContext())
  {
    var queryable = this.CreateAndInitializeQuery(context);

    queryable = this.AddContentPredicates(queryable, query);
    queryable = this.AddFacets(queryable, query);
    queryable = this.AddPaging(queryable, query);
    var results = queryable.GetResults();
    return this.SearchResultsFactory.Create(results, query);
  }
}

The method goes through all the IndexingProviders and grabs the search predicates (by calling GetQueryPredicate). If a Foundation or a Feature project has searchable content, then there is an IndexingProviders.

 
private IQueryable<SearchResultItem> AddContentPredicates(IQueryable<SearchResultItem> queryable, IQuery query)
{
  var contentPredicates = PredicateBuilder.False<SearchResultItem>();
  foreach (var provider in IndexingProviderRepository.QueryPredicateProviders)
  {
    contentPredicates = contentPredicates.Or(provider.GetQueryPredicate(query));
  }
  return queryable.Where(contentPredicates);
}

For instance in the Sitecore.Feature.PageContent project you will find a PageContentIndexingProvider, in Sitecore.Feature.News you will find NewsIndexingProvider etc.

Lets take a look at the PageContentIndexingProvider in the Sitecore.Feature.PageContent project, at method GetQueryPredicate. Notice the indexed fields which are placed in the fieldNames array.

  
public Expression<Func<SearchResultItem, bool>> GetQueryPredicate(IQuery query)
{
  var fieldNames = new[] {Templates.HasPageContent.Fields.Title_FieldName, Templates.HasPageContent.Fields.Summary_FieldName, Templates.HasPageContent.Fields.Body_FieldName};
  return GetFreeTextPredicateService.GetFreeTextPredicate(fieldNames, query);
}

Here are the fields in the Templates class:

public const string Title_FieldName = "Title";
public const string Summary_FieldName = "Summary";
public const string Body_FieldName = "Body";

Finally in GetFreeTextPredicateService.cs the search predicates will be built:

 
public Expression<Func<SearchResultItem, bool>> GetFreeTextPredicate(string[] fieldNames, IQuery query)
{
  var predicate = PredicateBuilder.False<SearchResultItem>();
  if (string.IsNullOrWhiteSpace(query.QueryText))
  {
    return predicate;
  }
  foreach (var name in fieldNames)
  {
    predicate = predicate.Or(i => i[name].Contains(query.QueryText));
  }
  return predicate;
}

So now we know how to figure out the names of the indexed fields for the PageContent.
To locate the rest of the indexed fields just identify the Indexingprovider in each projects(NewsIndexingProvider, PersonIndexingProvider and LocalDatasourceIndexingProvider).

We can do this in two ways, code change or update the schema.xml files.

Option 1 – Make code changes.
If we go for this option we will have to change the indexed field names for the IndexingProviders. Instead of the “Lucene” field names we will use “Solr” field names(in method GetQueryPredicate). Update the Templates.cs with the new field names:

public const string Title_FieldName = "title_t";
public const string Summary_FieldName = "summary_t";
public const string Body_FieldName = "body_t";

Do the same for rest of the “Lucene” fields in your habitat solution.

Now the search predicate builder will work just fine 🙂

Option 2 – Update the schema.xml files in each core.
With this approach we don’t need to do any code changes but we need to update the schema.xml in each core. Lets take a look at the schema.xml in core sitecore_web_index, locate the fields:

<field name="_content" type="text_general" indexed="true" stored="false" />
<field name="_database" type="string" indexed="true" stored="true" />
<field name="_path" type="string" indexed="true" stored="true" multiValued="true" /> 
...

Right under the fields in the xml file, we need to create the new “Lucene” fields and then copy the Solr fields to them. Like this:

<field name="title" type="text_general" indexed="true" stored="true" />
<field name="summary" type="text_general" indexed="true" stored="true" />
<field name="body" type="text_general" indexed="true" stored="true" />

<copyField source="title_t" dest="title" />
<copyField source="summary_t" dest="summary" />
<copyField source="body_t" dest="body" />

Do the same for rest of the “Lucene” fields in your Habitat solution.

Now you will have a working search predicate 🙂

Next are the facets, in the search method(above) you will see it’s calling the AddFacets(queryable, query) method. It uses the facets values from sitecore, you will find them here:
/sitecore/system/Settings/Buckets/Facets/Foundation/Indexing/Content Type

Please update the field name value to: content_type_t
(It’s a virtual field, that means it will be processed in Solr at runtime)

3. Disable Lucene config files and enable Solr config files

The final step is to to disable all the Lucene configs files and enable the Solr config files in your website folder. You can of course do it manually but…
Anders Laub mentions in his repository Docker containers for Sitecore development, about setting up SOLR in Habitat for an automated approach. The post he is referring to is great.
Instead of doing the job manually we will use a Powershell script and it will be triggered by a gulp task.

Here is the Powershell script:

function Set-SCSearchProvider($rootPath)
{
    $validInput = $true;
    #test that path is valid
    If (!(Test-Path -Path $rootPath))
    {
        Write-Host "The supplied path was invalid or inaccessible." -ForegroundColor Red;
        $validInput = $false;
    }
    If ($validInput)
    {
        Write-Host "Set to Solr." -ForegroundColor Yellow;
        $selectedProvider = "Solr";
        $deselectedProvider = "Lucene";
        #enumerate all config files to be enabled
        $filter = "*" + $selectedProvider + "*.config*";
        $filesToEnable = Get-ChildItem -Recurse -File -Path $rootPath -Filter $filter;
        foreach ($file in $filesToEnable)
        {
            Write-Host $file.Name;
            if (($file.Extension -ne ".config"))
            {
                $newFileName = [io.path]::GetFileNameWithoutExtension($file.FullName);
                $newFile = Rename-Item -Path $file.FullName -NewName $newFileName -PassThru;
                Write-Host "-> " $newFile.Name -ForegroundColor Green;
            }
        }
        #enumerate all config files to be disabled
        $filter = "*" + $deselectedProvider + "*.config*";
        $filesToDisable = Get-ChildItem -Recurse -File -Path $rootPath -Filter $filter;
        foreach ($file in $filesToDisable)
        {
            Write-Host $file.Name;
            if ($file.Extension -eq ".config")
            {
                $newFileName = $file.Name + ".disabled";
                $newFile = Rename-Item -Path $file.FullName -NewName $newFileName -PassThru;
                Write-Host "-> " $newFile.Name -ForegroundColor Green;
            }
        }
    }
}

Lets’s put it in the Configuration folder in your Habitat solution.

Here is the gulp script task that executes the Powershell script:

var exec = require("child_process").exec;
gulp.task("Setup-Solr-Config", function (callback) {
  
  exec("Powershell.exe -executionpolicy remotesigned . .\\setup-solr.ps1; Set-SCSearchProvider -rootPath '" + config.websiteRoot + "'",
    function (err, stdout, stderr) {
      console.log(stdout);
      callback(err);
    });
});

We will put it in our gulpfile.js.

Finally we are done.

Please try this and let me know if you have any issues/questions.

Don’t forget to rebuild your indexes in Sitecore.

That’s all for now folks 🙂