Each of the three search engine plugins (SqlSearch, Solr and ElasticSearch) buffer data while indexing records and write out to the engine when a threshold has been reached. The default for that threshold has been hardcoded into the plugin, and the only way to override it is to set the option directly on the plugin, something which requires code modification.
The buffer size has a direct impact on performance, memory usage and reliability. The larger the value, in general, the better the performance – at the expense of memory usage. Recently it has been observed that large values can cause index corruption with certain (definitely not all) SOLR installations.
To allow for local tweaking we need to make the buffer size configurable from search.conf. To do this we will implement a max_indexing_buffer_size directive.