Sitecore Rockz: Solr Configuration in SitecorePre-requisites:Site...

Sitecore Rockz: Solr Configuration in SitecorePre-requisites:
Site...: Solr Configuration in Sitecore Pre-requisites: Sitecore.NET 7.1 (rev. 140130) installed in the system Solr-4.1.0. Creating a S...

Tips and Tricks for Sitecore optimization

Publishing 

• How does Sitecore publish of millions of items? 

The simple answer is: nothing special is done. The rationale is in the answer to this question:
“Why would you need to publish 1 million items all the time?” The first time you publish from
authoring to delivery and if you were to use the Full Publish option, then the answer is: “Yes,
this would take some time.” But after that, you only need to run publish added content
incrementally.

• If I publish a single item in a bucket, but its bucket folders are not published, what happens? 

Sitecore has added an extra pipeline step to the publishing process to detect if an item
requires its bucket folders to be published and will add them to the publishing queue as well
automatically. This also applies to items that are in workflow. There is no need to add in an
extra workflow action for this.

• What publishing should I be doing for Item Buckets? 

The answer to this question does not change because of Item Buckets.

Standard Web.Config Tweaks 

• You should periodically tweak the cache depending on how many items are in the content 

tree and how many similar searches have been processed.

Setup Tweaks 

• When you import a lot of content programmatically, you must truncate the PublishingQueue, History, and Event Queue tables in the Master and Web databases and rebuild the indexes on the database tables. If you don’t do this, the PublishingQueue, History, and EventQueue tables will get very large, slow down processing, and your Sitecore installation may not start. After clearing the tables, you must rebuild the index and run a smart publish instead of an incremental publish. 

Environment Tweaks

• If possible,e, disable the inbuilt Windows Search Index as well as any other indexer that is running on the computer that runs the index or on the web server itself. This index uses essential Disk I/O resources that Lucene.net needs. 
• Don’t run processes on the index to create a backup. The index should not be part of regular backup procedures. Chances are that the backup will be outdated if it is ever needed, so it is a waste of resources. 
• It is very important that you set up a SQL Maintenance plan that rebuilds your indexes. When you create a lot of content, index fragmentation will increase, especially with the bulk importation of content. The hotspots will be the Items, Versioned, Unversioned, Shared, Blobs, and Links tables. To be on the safe side, you should set rebuilds for every table. If you don’t do this, performance of the CMS will degrade. 

Importing Data Tweaks 

• When you import a lot of content into Sitecore, it is best to use the BulkUpdateContext class. When you have imported the content, rebuild the Lucene index. 
• If you import a lot of content, do it in batches of, for example, one thousand, and then bucket or re-sync the bucket to avoid overloading the process with items. 

Uploading Files to the Media Library 

• The media library now supports the indexing of all files that support IFilter. For more information about IFilter, see http://en.wikipedia.org/wiki/IFilter. 
• Note :Sitecore does not by default provide search in PDF documents on MSSQL databases, and in neither PDF nor Word documents on Oracle databases.
• In short, IFIlter is a generic interface for indexing documents. Sitecore 7 ships configured to use IFilters to index text in the binary content of media items. To use this feature, you must install IFilters for the types of media items that you want your solution to index. You can use software such as the free IFilter Explorer from Citeknet to investigate the IFilters installed on your system. If the system hosting a Sitecore solution does not have an IFilter for a given media type, Sitecore can only index the metadata stored in that media item, not its binary content. Additionally, whether search results include media items can depend on the encoding of the format of data contained in those media. For example, IFilters may not be able to convert images of text in media items to structured text to parse. 
• Finally, you must install IFilters on the relevant hosts in your production environments (both content management and delivery); having an IFilter installed in a development environment will not allow indexing of that data type in your production environments. 

Bucket Config Tweaks 

• You can tweak your index so that it doesn’t index certain things that you don’t want in the index. This will decrease rebuild time and improve search time. 
• Consider rebuilding your indexes on a computer that has a solid state disk. Incremental updates do not have to be performed on SSD but they will benefit from this as well. If you have one dedicated server that rebuilds indexes and deploys them to an environment, ensure that this server has an SSD. Indexes will not be so big, so a small SSD will suffice — for example 64GB. 
• Don’t shard too many indexes. Sitecore must context switch between these shards and this slows down search time. 
• If you have very large caches, you can see large memory spikes when you run a search. This is normal as a search is filling the ItemCache for the results. Be careful of under-optimized 

Ref : Developer's Guide to Item Buckets and Search

Advantages of using Item Bucket? 

Applicable for Sitecore 7.0 and above.

An item bucket addresses the problem of managing large numbers of items within the content tree,
retrieving them, and working with them in a speedy and efficient manner. To decide if you should turn
an item into an item bucket, and in-turn, hide all its descendants, you must ask yourself if you care
about the structure of the data that is stored in the item bucket.

For example, if you have items within the content tree that contain a large number of sub items such
as products, media, or tags, it may be an advantage to turn these items into item buckets and thereby
remove the need for hierarchically managing the content.

Important
Note that a connection between two or more items does not necessarily need to be hierarchically.
Then when you want to work with a particular item placed in an item bucket, you can search for it and
open it. The advanced Sitecore search functionality allows you to search among all items in Sitecore
using for example free text, search filters, or facets, that makes it easier for you to find exactly what
you need.

Ref : http://sdn.sitecore.net/upload/sitecore7/72/developer's_guide_to_item_buckets_and%20search_sc72_a4.pdf 

Solr Configuration in Sitecore

Pre-requisites:

Sitecore.NET 7.1 (rev. 140130) installed in the system

Solr-4.1.0.

Creating a Solr Core

You need to create a Solr core to store all your index data. For example, the data collected from

Sitecore content items.

To create a Solr core:

1. In Solr, navigate to the solr directory.

Your root Solr installation should have a directory structure that looks similar to this:

2. Create an itembuckets folder in the Solr root. In this example no other cores are present.

Note

The name of the core does not have to be itembuckets. If you give it a different name, ensure that you update the index references in the file Sitecore.ContentSearch.Solr.Indexes.config (as explained in the next step).

3. Open the solr.xml file and update the core reference by entering the name of the home

Directory for the Solr core. In this example, the core is called item buckets.

<cores

 adminPath="/admin/cores"

 defaultCoreName="itembuckets"

 host="${host:}"

 hostPort="${jetty.port:}"

 hostContext="${hostContext:}"

 zkClientTimeout="${zkClientTimeout:15000}">

 <core name="itembuckets" instanceDir="itembuckets" />

</cores>

The itembuckets folder is now the home directory for this Solr core.

4. In the itembuckets folder, create the following three folders or sub directories:

Directory	Description
conf/	This directory is mandatory and must contain your solrconfig.xml and schema.xml. Any other optional configuration files are also stored here. You can find an example solrconfig.xml included in your Solr distribution.
data/	This directory is the default location for your index, and is used by the replication scripts for dealing with snapshots. You can override this location in the conf/solrconfig.xml. Solr creates this directory if it does not already exist.
lib/	This directory is optional. If it exists, Solr loads any Jars found in this directory and uses them to resolve any plugins specified in the solrconfig.xml or schema.xml. For example, Analyzers or Request Handlers. Alternatively you can use the <lib> syntax in conf/solrconfig.xml to direct Solr to your plugins. See the example conf/solrconfig.xml file for details.

Generating an XML Schema for Solr

The main difference between the default Lucene instance and Solr is that Solr needs a defined xml

schema when working with documents.

You can modify an existing schema using the Sitecore Build Solr Schema tool. This tool automatically

generates a basic schema and ensures all the fields that Sitecore needs are present. You can add

your own fields to this schema as long as you do not change the system index fields.

To generate a new Solr schema.xml file:

1.       In the Sitecore Desktop, open the Control Panel. In the Control Panel, click Indexing.

2. Then click Generate the Solr Schema.xml file.

3. In the Build Schema window, in the Source field, enter the path to the Solr schema that you want to use, for example: C:\apache-solr-4.0.0\solr\itembuckets\conf\schema.xml

In the Target File field, enter the destination for the new schema file, for example the website root folder: C:\inetpub\wwwroot\<sitename>\Website\schema.xml

4. Click Generate.

When you click Generate this updates the schema.xml and adds all the necessary fields that Sitecore needs. When the tool has finished, copy the file to the conf directory in the itembuckets/conf folder. It must also be renamed to schema.xml.

Note: If you have any other field definitions, copy fields or dynamics fields configured in your schema they are overwritten by the schema generator. To preserve these fields, copy your original schema and merge it with the newly generated schema afterwards.

Enabling Solr Term Support

When you implement Solr with Sitecore you need to enable term support in the Solr search handler.

The term functionality is built into Solr but is disabled by default. To power the dropdowns in the UI you must enable the terms component.

Note: In Lucene term support is enabled by default.

To enable Solr term support:

1. In your Solr installation, open the solrconfig.xml file. This file is located in the conf folder along with the schema.xml file. The path to this file is something like this:

apache-solr-4.0.0\solr\itembuckets\conf

2. In the solrconfig.xml file, locate the requestHandler node:

<requestHandler name="/select" class="solr.SearchHandler">

3. Inside this node you need to add the following line to the "defaults" node:

<lst name="defaults">

<str name="echoParams">explicit</str>

<int name="rows">10</int>

<str name="df">text</str>

<bool name="terms">true</bool>

</lst>

4. Also add the following node after the "defaults" element.

<arr name="last-components">

<str>terms</str>

</arr>

5. Save your changes.

Verifying that Solr is Running Correctly

After generating a new schema.xml file and updating the solrconfig.xml file you need to verify that Solr runs correctly.

To check Solr starts up correctly:              

1. Ensure Solr is not running. A full restart is needed so that it loads the new configuration.

2. Start up Solr.

Note

How you start up Solr is dependent on the type of installation you have.

3. Check the output log files. If there are no errors, then open the Solr administration page.

To open the administration page, enter the following URL in your browser: http://localhost:8983/solr/admin.

4. If you can see the Solr administration page and do not get any errors in the log files, then Solr is running correctly.

Configuring an IOC Container

The Sitecore Solr provider makes use of an IOC (Inversion of Control) container so that all the elements inside it are swappable without the need for re-compilation.

Sitecore supports five of the most commonly used open source IOC containers, all of which are currently available on the NuGet website.

Versions currently supported:

·         Castle Windsor v3.1.0.0

·         AutoFac v2.5.2

·         Ninject v3.0.0

·         StructureMap v2.6.2

·         Unity v2.1.505

Note

As these projects are open source they are often updated. You may have to request a specific version of the container from NuGet using the –Version switch in the NuGet command line.

Selecting the Correct Support DLL files

When you have chosen a suitable IOC container ensure that you include the correct support DLLs in the bin directory alongside the DLLs installed for the container.

Depending on your container, ensure that you also copy the following DLL files into the bin folder

Here we used the Castle Windsor IOC Container

1.       For Castle Windsor, copy the following files over to the project bin directory:

Castle.Facilities.SolrNetIntegration.dll 

Microsoft.Practices.ServiceLocation.dll 

Sitecore.ContentSearch.Linq.Solr.dll 

Sitecore.ContentSearch.SolrProvider.CastleWindsorIntegration.dll

Sitecore.ContentSearch.SolrProvider.dll

SolrNet.dll

2.       It is also necessary to add Castle.Core and Castle.Windsor.  I used version 3.1.0 for each.  Getting these is tricky.  You can create a solution and use NuGet, or you can pull them directly from the Nuget site, using https://www.nuget.org/api/v2/package/castle.windsor/3.1.0 and https://www.nuget.org/api/v2/package/castle.core/3.1.0  Hitting these URLs on Chrome automatically downloads a .nupkg object, which you can rename to a zip archive.  Both archives contain a "lib\net40-client" path.  Copy Castle.Windsor.dll and Castle.Core.dll from lib\net40-client of each package to the website bin directory.

Configuring Sitecore to work with Solr

Follow the steps in this section to configure Sitecore to work with Solr.

Important

Before you can enable the Solr config file you must copy the Sitecore.ContentSearch.Solr.Indexes.config file from your Sitecore Solr Support zip package to your website Include folder: wwwroot\<sitename>\App_Config\Include\

Enabling the Solr Config File

Your website Include folder contains several configuration files. Lucene search is enabled by default. If you want to use Solr you must disable the Lucene search config files and enable the Solr config file. This enables Solr integration and gives you access to all the Solr specific configuration settings.

To switch configuration files so that Solr is enabled and Lucene is disabled:

1. Navigate to the website Include folder: wwwroot\<sitename>\App_Config\Include\

Disable the following Lucene configuration files by adding .example to the file name extension.

2.     Enable the Sitecore.ContentSearch.Solr.Indexes.config file by removing .example from the file name.

Note

It is important to remove, or rename, the Lucene configuration include files so that they do not get loaded. If these files are not removed you get an error message. For more information on handling this error message, see the Troubleshooting section.

Solr Specific Settings

The following Solr specific settings can be found in the Sitecore.ContentSearch.Solr.Indexes.config file.

Specifying a Solr Service Address

This setting tells Sitecore where the Solr server is located. Sitecore appends the core name so only the base address needs to be supplied.

<setting name="ContentSearch.Solr.ServiceBaseAddress" value="http://localhost:8983/solr" />

Enabling a Search Provider

This setting tells Sitecore that Solr is enabled and so attempts to connect to the Solr server the next time the index is accessed. If it cannot connect you get an error. To disable, set this back to Lucene, which was the default setting.

<setting name="ContentSearch.Provider" value="Solr" /> (Default: “Lucene”)

Maximum Number of Search Results

This is a global setting found in the Sitecore.ContentSearch.config file.

This setting contains the maximum number of documents to retrieve on a single request if a limit has not been specified in the query, for example, Take(10). It is important to remember, for performance reasons, when querying how many results will be returned from the query being run and to handle them correctly, for example by using paging.

<setting name="ContentSearch.SearchMaxResults" value="500" />

Enabling Batch Mode :When an item is indexed the composed document is saved to the search index. When the default Lucene provider is enabled then each write is being flushed to a file on the local disk. When a document is written using the Solr provider the update has to travel over a network.

When an index is rebuilt a large number of document updates are created, this could result in a lot of network traffic which is not very efficient. Therefore using batch can help to optimize the update process as your indexes grow in size.

<setting name="ContentSearch.Update.BatchModeEnabled" value="true" />

<setting name="ContentSearch.Update.BatchSize" value="500" />

Batch mode (enabled by default) takes these document updates and only flushes to the Solr server when the batch has reached a certain size.

As your index grows you may want to increase this batch size to gain the most out of this process.

Specifying an IOC Container

The final part is to update the global.asax file so that the Solr provider is loaded on application start. You can do this by instructing your application to inherit from one of the application classes provided; specific configuration is dependent on your IOC container choice.

For example, to update the Global.asax file to use Castle Windsor:

1. In your website root folder, locate the Global.asax file:

wwwroot\<sitename>\Website

2. Open the Global.asax file and in the first line, replace the following:

Inherits="Sitecore.Web.Application"

With

Inherits="Sitecore.ContentSearch.SolrProvider.CastleWindsorIntegration.WindsorApplication"

This registers the IOC (inversion of control) components for Castle Windsor enabling Solr integration to work correctly.

For Castle Windsor IOC container

<%@Application Language='C#' Inherits="Sitecore.ContentSearch.SolrProvider.CastleWindsorIntegration.WindsorApplication" %>

Re-building the Search Indexes

Before you can start using the Solr indexing system you need to re-index all your Sitecore content.

To rebuild the Sitecore search indexes:

1. Log into the Sitecore Desktop.

2. Click Start and then click Control Panel.

3. In Control Panel, select Indexing. 

4. Then select Indexing Manager. 

5. In the Select Search Index window, select all local indexes.

6. Click Rebuild and then click Finish when the Wizard has completed.

When the Wizard has finished, try to perform a search in the Sitecore Content Editor using the item buckets search functionality.

Cheers. 

Please provide your valuable comments.

Creating a New Search Facet in Sitecore

You can use facets to drill down to more specific results in any list of search results. The default facets are displayed in the facets menu on the right side of the search results. To create a custom facet, navigate to the /sitecore/system/Settings/Buckets/Facets item of the content tree. Right click on the Facets item and in the context menu, click Insert, Facet. You now have to specify the name of the field in your index, in the parameters field in the content tab. You can apply hierarchical faceting by listing many fields separated by commas. This is useful if you want to facet on, for example, Clothes Type first, and then on Color...

Example

var results = queryable.FacetOn(d => d.Name);
var facets = results.GetFacets();
foreach (var category in facets.Categories)
{
    Console.WriteLine(category .Name);
    foreach (var facetValue in category.Values)
    {
         Console.WriteLine("{0}: {1}", facetValue.Name, facetValue.Aggregate);
    }
}

Getting Sitecore items

All items ca be accessed from Sitecore.Data.Items.Item class.

//Get the master database through a Factory
Database masterDb = Sitecore.Configuration.Factory.GetDatabase("master");



//Retrieving an item with a path through the database
Sitecore.Data.Items.Item myItem = masterDb.Items["/sitecore/content/home"];


//Retrieving an item from an item id
Sitecore.Data.Items.Item mySecondItem = masterDb.Items[new ID("{CC07A18cA-670D-5458-9A1S-G3072D1A0BE5}")];

Advantages of using sublayouts:

Advantages of using sublayouts:

• Sublayouts support a superset of the features that XSL renderings, web controls, and user controls support.
• Sublayouts provide markup templates with code-behind to separate logic from presentation.
• Sublayouts can access all Sitecore APIs and any other APIs available to the ASP.NET worker process.
• Sublayouts support placeholders, allowing dynamic nesting of presentation components to any level.
• Sublayouts support debugging with the Visual Studio debugger.
• Sublayouts support compile-time error detection.
• If it helps in the migration of existing code or the integration of third-party components, you can easily implement a sublayout as a wrapper for a single web control.