Friday, August 7, 2020

Indexing Spatial data(Multipolygons) on Solr

Note: Good as of v8.6


  • Copy JTS jar to SOLR_INSTALL/server/solr-webapp/webapp/WEB-INF/lib


  • Update location_rpt field type on schema xml to:

        <fieldType name="location_rpt"
            class="solr.SpatialRecursivePrefixTreeFieldType"
            autoIndex="true"
            validationRule="repairBuffer0"
            spatialContextFactory="JTS"
            geo="true" distErrPct="0.025" maxDistErr="0.001"
            distanceUnits="degrees" />

  • change the geometry field's data type to location_rpt and make it multivalued
  • To index, assign WKT value to the geometry field.
  • To query for all polygons within a bounding box:

q    *:*
fq        geometry:[25,-80 TO 26,-81]
lower-left corner as the start of the range and the upper-right corner as the end of the range.


Monday, May 11, 2020

Solr DIH with pgsql

Configuring Solr DataImportHandler with postgres (Good as of Solr 8.5.1)


  • Copy the database jdbc driver jar to ${solr.install.dir:../../../..}/contrib/dataimporthandler-extras/lib



  • Add the following lines to solrconfig.xml of the core which needs DIH enabled:


// To allow new fields to be added to schema automatically
  <schemaFactory class="ManagedIndexSchemaFactory">
   ​ <bool name="mutable">true</bool>
    ​<str name="managedSchemaResourceName">managed-schema</str>
  ​</schemaFactory>
//Add DIH jars to classpath
<lib dir="${solr.install.dir:../../../..}/contrib/dataimporthandler-extras/lib" regex=".*\.jar" />
        <lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-dataimporthandler-.*\.jar" />
// Provide config file name for data import
<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
  <lst name="defaults">
    <str name="config">data-config.xml</str>
  </lst>
</requestHandler>


  • Modify the following section to make sure it adds only single valued fields by default:


  <updateProcessor class="solr.AddSchemaFieldsUpdateProcessorFactory" name="add-schema-fields">
    <lst name="typeMapping">
      <str name="valueClass">java.lang.String</str>
      <str name="fieldType">string</str>
      <lst name="copyField">
        <str name="dest">*_str</str>
        <int name="maxChars">256</int>
      </lst>
      <!-- Use as default mapping instead of defaultFieldType -->
      <bool name="default">true</bool>
    </lst>
    <lst name="typeMapping">
      <str name="valueClass">java.lang.Boolean</str>
      <str name="fieldType">boolean</str>
    </lst>
    <lst name="typeMapping">
      <str name="valueClass">java.util.Date</str>
      <str name="fieldType">pdate</str>
    </lst>
    <lst name="typeMapping">
      <str name="valueClass">java.lang.Long</str>
      <str name="valueClass">java.lang.Integer</str>
      <str name="fieldType">plong</str>
    </lst>
    <lst name="typeMapping">
      <str name="valueClass">java.lang.Number</str>
      <str name="fieldType">pdouble</str>
    </lst>
  </updateProcessor>

 

Update/Create your data-config.xml. Here's a sample:

 <dataConfig>
  <dataSource type="JdbcDataSource" name="db-name" driver="org.postgresql.Driver" url="jdbc:postgresql://server-hostname:5432/db-name" user="uname" password="pword" />
  <document name="layer_features">
    <entity name="name-on-solr-admin-dropdown" query="select 'aa2c4f3f-b600-4ee9-9997-f030b4e1f4de' as layers_id,ST_AsText(z.geom) as geometry, c.* from zip c, florida_zip_simplified z where c.zip=z.&quot;ZIP&quot; and c.countyname=z.&quot;COUNTYNAME&quot;;
" collection="layer_features">
    <field column="countyname" name="countyname"/>
    <field column="poname" name="poname"/>
    <field column="depcode" name="depcode"/>
    <field column="zip" name="zip"/>
    <field column="cases_1" name="casesall"/>
    <field column="geometry" name="geometry"/>
    <field column="layers_id" name="layers_id"/>   
    </entity>
  </document>
</dataConfig>

Note: column is the name from database, name is the field name on solr schema.
  • Restart Solr

Monday, May 20, 2019

Install SSL certificate on CentOS7/httpd

Note: This is just a documentation for myself and not a reference material

1) Apply for security certificate:
University students, faculty, staff & departments can request InCommon Certificates.  

- To apply for your InCommon SSL certificate, please go to: https://cert-manager.com/customer/InCommon/ssl?action=enroll.
- Our Access Code is: 3565, then enter your email address.
- Your email address must be an @miami or @umiami.edu address.
- Please fill out the information requested. You may select Certificate Terms up to 3 years or what best suits your technical needs.
- You are able to reissue your certificate to a different server at any time. In order to do so, please send an email to software@miami.edu for instructions.
- Once you submit this form, the approval process can take 24-48 hours during Monday through Friday normal business hours. You will receive an email when it becomes active.
 
 
You will receive 3 certificates. 
Copy the X509 Certificate only and X509 Intermediates/root only certificates to /etc/pki/tls/certs/
 
On your CentOS 7 server, run the following commands:

openssl genrsa -out i2b2app-stage.ccs.miami.edu.key 2048
openssl req -new -key i2b2app-stage.ccs.miami.edu.key -out i2b2app-stage.ccs.miami.edu.csr
vi i2b2app-stage.ccs.miami.edu.csr ( Enter this CSR on the request form)
cp i2b2app-stage.ccs.miami.edu.key /etc/pki/tls/private/
cp i2b2app-stage.ccs.miami.edu.csr /etc/pki/tls/certs/
openssl x509 -in i2b2app-stage.ccs.miami.edu.csr -text
openssl x509 -in ../../tls/private/i2b2app-stage.ccs.miami.edu.key -text 
yum install openssl mod_ssl
chkconfig --level 345 httpd on
/usr/sbin/setsebool -P httpd_can_network_connect 1
vi /etc/httpd/conf.d/ssl.conf
 

Add this VirtualHost section to httpd's ssl.conf:

##
## SSL Virtual Host Context
##
<VirtualHost i2b2app-stage.ccs.miami.edu:80>
     ServerName i2b2app-stage.ccs.miami.edu:80
     Redirect / https://i2b2app-stage.ccs.miami.edu/
</VirtualHost>
ProxyRequests off

<VirtualHost i2b2app-stage.ccs.miami.edu:443>

# General setup for the virtual host, inherited from global configuration
#DocumentRoot "/var/www/html"
ServerName i2b2app-stage.ccs.miami.edu:443

# Use separate log files for the SSL virtual host; note that LogLevel
# is not inherited from httpd.conf.
ErrorLog logs/ssl_error_log
TransferLog logs/ssl_access_log
LogLevel warn

#   SSL Engine Switch:
#   Enable/Disable SSL for this virtual host.
SSLEngine on

#   SSL Protocol support:
# List the enable protocol levels with which clients will be able to
# connect.  Disable SSLv2 access by default:
#SSLProtocol all -SSLv2
SSLProtocol -all +TLSv1.2 -SSLv2 -SSLv3

#   SSL Cipher Suite:
# List the ciphers that the client is permitted to negotiate.
# See the mod_ssl documentation for a complete list.
#SSLCipherSuite DEFAULT:!EXP:!SSLv2:!DES:!IDEA:!SEED:+3DES

SSLCipherSuite HIGH:!MEDIUM:!aNULL:!eNULL:!MD5:!RC2:!RC4:!DES:!3DES:!IDEA:!EXP:!EXPORT:!SEED
SSLHonorCipherOrder on

#   Server Certificate:
# Point SSLCertificateFile at a PEM encoded certificate.  If
# the certificate is encrypted, then you will be prompted for a
# pass phrase.  Note that a kill -HUP will prompt again.  A new
# certificate can be generated using the genkey(1) command.
SSLCertificateFile /etc/pki/tls/certs/i2b2app-stage_ccs_miami_edu_cert.cer

#   Server Private Key:
#   If the key is not combined with the certificate, use this
#   directive to point at the key file.  Keep in mind that if
#   you've both a RSA and a DSA private key you can configure
#   both in parallel (to also allow the use of DSA ciphers, etc.)
SSLCertificateKeyFile /etc/pki/tls/private/i2b2app-stage.ccs.miami.edu.key

#   Server Certificate Chain:
#   Point SSLCertificateChainFile at a file containing the
#   concatenation of PEM encoded CA certificates which form the
#   certificate chain for the server certificate. Alternatively
#   the referenced file can be the same as SSLCertificateFile
#   when the CA certificates are directly appended to the server
#   certificate for convinience.
SSLCertificateChainFile /etc/pki/tls/certs/i2b2app-stage_ccs_miami_edu_interm.cer

#   Certificate Authority (CA):
#   Set the CA certificate verification path where to find CA
#   certificates for client authentication or alternatively one
#   huge file containing all of them (file must be PEM encoded)
#SSLCACertificateFile /etc/pki/tls/certs/ca-bundle.crt

#   Client Authentication (Type):
#   Client certificate verification type and depth.  Types are
#   none, optional, require and optional_no_ca.  Depth is a
#   number which specifies how deeply to verify the certificate
#   issuer chain before deciding the certificate is not valid.
#SSLVerifyClient require
#SSLVerifyDepth  10

#   Access Control:
#   With SSLRequire you can do per-directory access control based
#   on arbitrary complex boolean expressions containing server
#   variable checks and other lookup directives.  The syntax is a
#   mixture between C and Perl.  See the mod_ssl documentation
#   for more details.
#<Location />
#SSLRequire (    %{SSL_CIPHER} !~ m/^(EXP|NULL)/ \
#            and %{SSL_CLIENT_S_DN_O} eq "Snake Oil, Ltd." \
#            and %{SSL_CLIENT_S_DN_OU} in {"Staff", "CA", "Dev"} \
#            and %{TIME_WDAY} >= 1 and %{TIME_WDAY} <= 5 \
#            and %{TIME_HOUR} >= 8 and %{TIME_HOUR} <= 20       ) \
#           or %{REMOTE_ADDR} =~ m/^192\.76\.162\.[0-9]+$/
#</Location>

#   SSL Engine Options:
#   Set various options for the SSL engine.
#   o FakeBasicAuth:
#     Translate the client X.509 into a Basic Authorisation.  This means that
#     the standard Auth/DBMAuth methods can be used for access control.  The
#     user name is the `one line' version of the client's X.509 certificate.
#     Note that no password is obtained from the user. Every entry in the user
#     file needs this password: `xxj31ZMTZzkVA'.
#   o ExportCertData:
#     This exports two additional environment variables: SSL_CLIENT_CERT and
#     SSL_SERVER_CERT. These contain the PEM-encoded certificates of the
#     server (always existing) and the client (only existing when client
#     authentication is used). This can be used to import the certificates
#     into CGI scripts.
#   o StdEnvVars:
#     This exports the standard SSL/TLS related `SSL_*' environment variables.
#     Per default this exportation is switched off for performance reasons,
#     because the extraction step is an expensive operation and is usually
#     useless for serving static content. So one usually enables the
#     exportation for CGI and SSI requests only.
#   o StrictRequire:
#     This denies access when "SSLRequireSSL" or "SSLRequire" applied even
#     under a "Satisfy any" situation, i.e. when it applies access is denied
#     and no other module can change it.
#   o OptRenegotiate:
#     This enables optimized SSL connection renegotiation handling when SSL
#     directives are used in per-directory context.
#SSLOptions +FakeBasicAuth +ExportCertData +StrictRequire
<Files ~ "\.(cgi|shtml|phtml|php3?)$">

    SSLOptions +StdEnvVars
</Files>
<Directory "/var/www/cgi-bin">
    SSLOptions +StdEnvVars
</Directory>

#   SSL Protocol Adjustments:
#   The safe and default but still SSL/TLS standard compliant shutdown
#   approach is that mod_ssl sends the close notify alert but doesn't wait for
#   the close notify alert from client. When you need a different shutdown
#   approach you can use one of the following variables:
#   o ssl-unclean-shutdown:
#     This forces an unclean shutdown when the connection is closed, i.e. no
#     SSL close notify alert is send or allowed to received.  This violates
#     the SSL/TLS standard but is needed for some brain-dead browsers. Use
#     this when you receive I/O errors because of the standard approach where
#     mod_ssl sends the close notify alert.
#   o ssl-accurate-shutdown:
#     This forces an accurate shutdown when the connection is closed, i.e. a
#     SSL close notify alert is send and mod_ssl waits for the close notify
#     alert of the client. This is 100% SSL/TLS standard compliant, but in
#     practice often causes hanging connections with brain-dead browsers. Use
#     this only for browsers where you know that their SSL implementation
#     works correctly.
#   Notice: Most problems of broken clients are also related to the HTTP
#   keep-alive facility, so you usually additionally want to disable
#   keep-alive for those clients, too. Use variable "nokeepalive" for this.
#   Similarly, one has to force some clients to use HTTP/1.0 to workaround
#   their broken HTTP/1.1 implementation. Use variables "downgrade-1.0" and
#   "force-response-1.0" for this.
SetEnvIf User-Agent ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0

#   Per-Server Logging:
#   The home of a custom SSL log file. Use this when you want a
#   compact non-error SSL logfile on a virtual host basis.
CustomLog logs/ssl_request_log \
          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"


# Configure proxy and reverse proxy for i2b2 web services to JBoss localhost port
SSLProxyEngine on
ProxyPass /i2b2 http://localhost:9090/i2b2
ProxyPassReverse /i2b2 http://localhost:9090/i2b2

</VirtualHost>