The upgrade process for LucidWorks Search includes several steps to update the system configuration files, the search indexes, the LucidWorks software and any embedded software from third-party vendors (such as Solr and Lucene, etc.).
It's recommended that the process described below be performed on a test system before upgrading your production application.
Supported Versions
The upgrade process described in this section only applies to upgrades within v2.x. If you are using v1.7 or v1.8 of LucidWorks, and would like to move to v2.1, you will first need to move to v2.0. Contact LucidWorks Support for help with an upgrade from v1.x to v2.x.
If you are using v2.0.x, you can upgrade directly to v2.1.x or v2.5.x.
In version 2.0 of LucidWorks Search, we used a process for migrating from v1.7 or v1.8 to v2.0 that used the LucidWorks REST API to copy configuration information from the older version of LucidWorks and import it into the new version of LucidWorks. That process was been completely replaced in v2.1. The current process upgrades the older version in place. A parallel installation of LucidWorks is made to get the new code, then a script is run to update the existing configuration files.
Upgrade Steps
Upgrading LucidWorks requires the following steps:
- Preparation: Review the Upgrade Notes
- Step 1: Stop Existing Installation and Create a Backup
- Step 2: Install the New Version of LucidWorks
- Step 3: Run the MetadataUpgradeTool
- Step 4: Upgrade the Indexes or Delete Them
- Step 5: Update the $LWE_HOME/app dir
- Step 6: Make Manual Changes to Configuration Files
- Step 7: Start LucidWorks
 | If you are upgrading to v2.5.3 from v2.5.2, do not run Steps 3 and 4 detailed below. |
Preparation: Review the Upgrade Notes
The following instructions contain step-by-step information about the upgrade process. There may, however, be specific changes for each release and for your specific implementation that need to be considered before, during, or after the upgrade. Before starting any upgrade, please review the list of changes for your version in the section Upgrade Notes.
Also note any manual changes you may have made to configuration files found in $LWE_HOME/conf/master.conf. You will need to re-apply those changes manually in Step 6: Make Manual Changes to Configuration Files, described below.
Step 1: Stop Existing Installation and Create a Backup
LucidWorks Search can be stopped with the scripts available in $LWE_HOME/app/bin. Use either stop.sh or stop.bat depending on your operating system.
A backup of your existing installation can be made by copying the entire $LWE_HOME directory to a parallel location. If your installation has been made in a directory called /usr/local/LucidWorksEnterprise, then in Unix, for example, the command would be:
cp -r /usr/local/LucidWorksEnterprise /tmp/lwe-old-backup
You can name the backup whatever you'd like, as long as you remember it later if you need it.
Now is also a good time to make note of any customizations that will be over-ridden by the upgrade, if you haven't done so already. For details of these changes, see the Upgrade Notes for your version.
Step 2: Install the New Version of LucidWorks
Follow the steps outlined in the sections on installing LucidWorks to install the new version of LucidWorks. This can be done in any location on your server, but you should change the default path if your existing LucidWorks is already installed there. You can choose the same ports as are used in your existing installation: this installation will only be used to get an updated app directory, the MetadataUpgradeTool, and the IndexUpgradeTool (See Step 3 and Step 4 below).
| Do not install the new version of LucidWorks Search over your existing installation! |
When the installer asks if you would like to start the new version of LucidWorks, say no (uncheck the box).
The instructions from here on will refer to the top level of this installation as $LWE_NEW.
Step 3: Run the MetadataUpgradeTool
| If you are upgrading to v2.5.3 from v2.5.2, do not run this step. Skip to [#Step 5: Update the {{$LWE_HOME/app}} dir]. There are no configuration changes that need to be made with this tool.
If you are using v2.1.1 or earlier, you should complete this step. |
The MetadataUpgradeTool is found in the $LWE_NEW/app/migration_tools directory. It will update your configuration files to the proper format for the new version and will also move the user database from the app dir to the data dir if it still exists in the old location.
To run it, issue this command:
$ java -jar $LWE_NEW/app/migration_tools/MetadataUpgradeTool.jar -verbose -lwe_conf_dir <existing conf dir> -lwe_app_dir <existing app dir> -lwe_data_dir <existing data dir>
The parameters -lwe_conf_dir, -lwe_app_dir, and -lwe_data_dir are required and should point to the $LWE_HOME/conf, $LWE_HOME/app and $LWE_HOME/data directories, respectively, of your existing LucidWorks application.
The parameter -verbose is optional, but highly recommended. It allows the full output of the tool to show on-screen in case there are problems or failures. If support is needed after using the MetadataUpgradeTool, Lucid Imagination will request this output. The output will look something like:
### Upgrading format of collections.yml ... Writing upgraded collections.yml ### Upgrading similarity configuration in Solr schema ... Upgrading similarity configuration in /Applications/LucidImagination/lwe2.0.1-upgrade/conf/solr/cores/collection1_0/conf/schema.xml Writing upgraded /Applications/LucidImagination/lwe2.0.1-upgrade/conf/solr/cores/collection1_0/conf/schema.xml Upgrading similarity configuration in /Applications/LucidImagination/lwe2.0.1-upgrade/conf/solr/cores/LucidWorksLogs/conf/schema.xml Writing upgraded /Applications/LucidImagination/lwe2.0.1-upgrade/conf/solr/cores/LucidWorksLogs/conf/schema.xml ### Upgrading field mapping configuration in solrconfig ... Processing /Applications/LucidImagination/lwe2.0.1-upgrade/conf/solr/cores/collection1_0/conf/solrconfig.xml ... Writing upgraded /Applications/LucidImagination/lwe2.0.1-upgrade/conf/solr/cores/collection1_0/conf/solrconfig.xml Processing /Applications/LucidImagination/lwe2.0.1-upgrade/conf/solr/cores/LucidWorksLogs/conf/solrconfig.xml ... Writing upgraded /Applications/LucidImagination/lwe2.0.1-upgrade/conf/solr/cores/LucidWorksLogs/conf/solrconfig.xml ### Removing old autocomplete data files... ### Upgrading UI DB location... Moved /Applications/LucidImagination/lwe2.0.1-upgrade/app/webapps/lwe-ui/WEB-INF/db/production.sqlite3 to /Applications/LucidImagination/lwe2.0.1-upgrade/data/lwe-ui ### Upgrading UI DB... Executing UI DB upgrade == AddExternalProtocolToSettings: migrating ================================== -- change_table(:settings) -> 0.0060s == AddExternalProtocolToSettings: migrated (0.0070s) =========================
| If you have created any collection templates they will not be upgraded with the MetadataUpgradeTool. It's recommended that all templates created with previous versions be recreated from scratch using a new or upgraded collection. |
 | One of the changes from v2.0 to v2.1 was to change the names of the data source types "S3" and "S3N". The "S3N" type, for Amazon over S3, is now known as "S3". The former "S3" type, for a Hadoop Filesystem on S3, is now known as "S3H". During the upgrade process, these types will be converted automatically.
However, there is a new requirement for these types of data sources that the value for the URL must include a trailing slash if the URL points to a directory of files and not to a single resource. This is not done automatically by the upgrade. In order to successfully crawl an existing "S3" or "S3H" data source after upgrading, the trailing slash must be added manually if the path specified is not to a single file or resource. |
Step 4: Upgrade the Indexes or Delete Them
| If you are upgrading to v2.5.3 from v2.5.2, do not run this step. Skip to Step 5: Update the $LWE_HOME/app dir. There are no index changes that need to be made with this tool, you can use them as is.
If you are using v2.1.1 or earlier, you should complete this step. |
The MetadataUpgradeTool only upgrades configuration and other system files. However, the search indexes also need to be upgraded as a separate step. If, however, the indexed content is not needed, the indexes can be simply deleted instead of upgraded.
Upgrading the Indexes
In order to upgrade an index from an earlier version of LucidWorks, the Index Upgrade Tool should be run. This tool is found under $LWE_HOME/app/migration-tools/ in a .jar file called IndexUpgradeTool.jar.
| It is recommended to run this Upgrade Tool only after consultation with Lucid Imagination Support to be sure you understand the full ramifications of running this tool on your local, customized, index. |
While we have provided this index upgrade tool to help you avoid re-indexing your data, it is recommended to do so with every migration to a new version.
Before running the tool, you should make sure LucidWorks is not running to ensure there are no indexing processes taking place while performing the upgrade.
To run the tool, open a command line interface, navigate to the $LWE_NEW/app/migration_tools directory and issue the command:
$ java -jar IndexUpgradeTool.jar [options] <sourcedir> <destinationdir>
The <sourcedir> parameter is the existing index that will be upgraded and moved to the <destinationdir>. Unlike the MetadataUpgradeTool, which updates configuration files in place, the IndexUpgradeTool makes a copy of the index while upgrading it. The <sourcedir> is $LWE_HOME/data/solr/cores/collection/data/index (where collection is the name of the collection to be upgraded. The destinationdir should be a temporary location (preferably a directory, such as /tmp/index-upgrade, as the output is a number of raw index files).
 | The IndexUpgradeTool can upgrade the index for one collection at a time. If there are multiple collection in the origin LucidWorks installation, these would each need to be converted separately. By default, there are at least two indexes: the one that contains your data (called collection1 at initial installation) and one that indexes log data called LucidWorksLogs. Don't forget to upgrade the LucidWorksLogs collection or delete the index - skipping that collection will cause LucidWorks to fail on restart. |
There is currently one option that can be used with the IndexUpgradeTool, which is -checkindex. This will run Lucene's checkindex tool to validate that the index is correct. This is a good idea to do, especially for systems already in production, but will add time to the upgrade process.
The output of the tool is a number of index files in the location you specified with the <destinationdir> parameter. Once the tool has finished, delete the existing index files for each collection (found under $LWE_HOME/data/solr/cores/collection/data/index) and copy the new index files for each collection to the index dir.
Deleting the Indexes
If the index files are not particularly needed, which may be the case for the LucidWorksLogs collection particularly, they can be deleted. To do so, delete all of the subdirectories found under $LWE_HOME/data/solr/cores/collection/ for each collection.
 | If you are using the DirectSpellChecker, which is the LucidWorks Search default spell check implementation, your spell check data will be recreated automatically with either index upgrade method described above (i.e., either by re-indexing all content or upgrading the index with the IndexUpgradeTool). This is because the DirectSpellChecker uses the main index to create spelling suggestions.
If, however, you have changed the default implementation and are using the IndexBasedSpellChecker, your spell check data is not incorporated with the main index, but instead stored as a separate index. In this case, these indexes will not be upgraded with the IndexUpgradeTool (nor deleted by deleting the main index) and may cause problems when restarting LucidWorks. If possible, these indexes should be deleted and rebuilt after the main index has been upgraded. |
Step 5: Update the $LWE_HOME/app dir
Neither the MetadataUpgradeTool nor the IndexUpgradeTool update the program files that make up the LucidWorks application. These need to be updated manually by replacing the app directory in the original installation with the app directory from the new LucidWorks Search installation. To do this, delete the original $LWE_HOME/app and copy the $LWE_NEW/app directory to $LWE_HOME. Assuming default locations of the installations, the sequence would be:
$ rm -r /LucidWorks/LucidWorksSearch/app $ cp -r /LucidWorks/LWS-newVersion/app /LucidWorks/LucidWorksSearch
Step 6: Make Manual Changes to Configuration Files
Update master.conf
The upgrade process does not modify the master.conf file found in $LWE_HOME/conf/. Two changes are recommended for optimal performance in typical situations. They are not added automatically by the MetadataUpgradeTool because these or similar changes may have already been made in your implementation depending on your specific situation. Both parameters are entered in the section of the file for JVM Settings for LWE-UI:
- Add -XX:MaxPermSize=256M
- Add -Dcom.sun.management.jmxremote
Depending on the 2.x version you started with, these parameters may already be correct.
If you have added any other changes to master.conf, such as adding the zkHost for SolrCloud startup or performance-related changes, you should note those changes in the original master.conf file and re-apply those to the new master.conf.
Other Changes
Other manual changes may be required depending on the version and your own implementation. Review the Upgrade Notes for your version for details.
Step 7: Start LucidWorks
Start LucidWorks Search using the scripts available in $LWE_HOME/app/bin. Use either start.sh or start.bat depending on your operating system.
| After the upgrade, the indexes that drive auto-complete will have been removed. The auto-complete indexes are not automatically generated unless there is already an auto-complete activity scheduled (and it will happen on its schedule). If using auto-complete in your search application, and it is not scheduled to run periodically, use the Activity API or the Admin UI to schedule it and recreate the auto-complete indexes. |
After checking to be sure the upgrade was successful, the $LWE_NEW installation and/or the backup can be deleted. In order to be sure the upgrade was successful, try to index some content and perform some queries, while checking for errors in the "core" log found in $LWE_HOME/data/logs.
