Skip to content

FAQ

Problems and resolutions

Adding 7-mode systems

Check that you enabled TLS with options tls.enable on. Also check the following options : httpd.admin.access, httpd.admin.hostsequiv.enable

Grafana Alerting

Grafana 4 supports alerting only for queries that do not make use of variables, so if you edit the panel and you see anything with a '$' sign in the query used to fetch the metrics, alerting will not work. What you can do, is create your own panels without tempate variables and use it to alert.

Network conflict (Docker)

It might happen that NAbox internal network for docker containers overlaps one of your own networks.

When that happens, you notice that you cannot get to NAbox web site from those overlapping networks.

To fix this, you need to log into NAbox with root, and edit /usr/local/nabox/docker-compose/docker-compose.override.yaml.

There is a sample configuration ready to use, you just have to un-comment the whole networks section like so :

# Use this file to customize docker-compose configuration.
# For example, you can change default network CIDR like this :
#
networks:                                                         
  default:                                                            
    driver: bridge                                               
    name: docker-compose_default                                         
    ipam:                                                                 
      driver: default 
      config:                                               
        - subnet: 172.98.0.0/16
          gateway: 172.98.0.1
version: "3.7"

Feel free to change subnetand gateway to values that suits you.

After doing modification to this file, you need to stop the containers :

# dc down
Stopping graphite           ... done
Stopping nabox-admin        ... done
Stopping nabox-harvest      ... done
Stopping nabox-api          ... done
Stopping nabox-harvest2     ... done
Stopping grafana            ... done
Stopping prometheus         ... done
Stopping traefik            ... done
Stopping container-exporter ... done
Stopping go-carbon          ... done
Removing graphite           ... done
Removing nabox-admin        ... done
Removing nabox-harvest      ... done
Removing nabox-api          ... done
Removing nabox-harvest2     ... done
Removing grafana            ... done
Removing prometheus         ... done
Removing traefik            ... done
Removing container-exporter ... done
Removing go-carbon          ... done
Removing network docker-compose_default

After a few seconds, NAbox will restart everything with the right network configuration.

Reset Network configuration

If you find yourself stuck in a situation where NAbox has no IP address configured, and you have to manually reconfigure from the console, you can use one of those two methods :

Simply edit /etc/network/interfaces, and use the following for static configuration : 

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0

iface eth0 inet static
address 192.168.0.101
netmask 255.255.255.0
gateway 192.168.0.1

Or this file if you use DHCP : 

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0

iface eth0 inet dhcp

Then restart network : 

service networking restart

There is a much more convoluted way to change IP configuration, indicated here more as a proof of concept/demonstration than for actual practical method : 

# For static IP configuration
curl -X POST -uadmin:Netapp01 -H "Content-type: application/json" -d '{
  "hostname": "nabox",
  "ip": {
    "dns": [
      "192.168.0.100"
    ],
    "domain": "company.com",
    "gateway": "192.168.0.1",
    "ip_address": "192.168.0.100",
    "netmask": "255.255.255.0"
  },
  "use_dhcp": false
}' http://`docker inspect nabox-api|jq -r '.[0].NetworkSettings.Networks["docker-compose_default"].IPAddress'`:5000/api/1.0/system/network-config

# For DHCP configuration
curl -X POST -uadmin:Netapp01 -H "Content-type: application/json" -d '{
  "hostname": "nabox",
  "use_dhcp": true
}' http://`docker inspect nabox-api|jq -r '.[0].NetworkSettings.Networks["docker-compose_default"].IPAddress'`:5000/api/1.0/system/network-config

Unable to add systems with internal Root CA

Due to a bug in NAbox, current versions will error if trying to add a system that has HTTPS with an internal root CA.

The fix is to edit /usr/local/nabox/docker-compose/docker-compose.override.yaml and set the following environment variable :

version: "3.7"
services:
  nabox-harvest2:
    environment:
      - REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

General questions

Migrating from older versions

While there is no direct upgrade path from NAbox 2.x to 3.x, there are options to make your migration easier.

Your fastest and easiest way to move to NAbox 3 is to run both NAbox side-by-side, and wait for NAbox 2.6 data to age out and be not relevant for you.

If you absolutely need to run a single virtual appliance, know that you can actually run Harvest 1.6 in NAbox 3, and have both Harvest 1.6 and Harvest 2.x running at the same time, the former storing into Graphite, the later into Prometheus.

Migrating configurations

Again, NAbox 3 does not directly import cluster configurations from NAbox 2, so it will have to be manual. You can leverage the full API provided with NAbox 3 to make the switch easier, for example using a script reading Harvest 1.6 configuration file, and executing REST calls to configure the systems into NAbox 3.

Here is an example to get you started : you can extract configuration of all the clusters configured into NAbox 2.x with this command line :

sed -E -n "/^\[/,/^$/{s/^[^=]*= *//g;H;/^$/{x;s/\n/,/g;p;x;h;};}" /opt/conf/harvest/netapp-harvest.conf |grep -E -v ",\[(global|default)\]"
,[AFF],aff8040,Paris,admin,sillyP@ssWd,60,default,cdot-policy-group.conf,
,[ClusterA],mcca,LAB,wfa,sillyP@ssWd,60,default,cdot-policy-group.conf,
,[ClusterB],mccb,LAB,wfa,sillyP@ssWd,60,default,cdot-policy-group.conf,

Root access

At your own risk, for troubleshooting or customizing purpose, you can access the virtual appliance as root with the following password : NetappGrafanaVA. You can change the root password from the web interface.

Reset password

If you're in a situation where you cannot login using admin account to NAbox web ui or Grafana, you might need to reset the admin password.

If you still have the root password which is NetappGrafanaVA by default, you can run the following command to reset admin password from NAbox CLI :

PASSWORD="yourNewP@ssW0rd"
curl -k -X PUT -H "api_key: `cat /opt/secrets/nabox-api`" -H "Content-type: application/json" -d "{\"new_password\":\"$PASSWORD\",\"update_root\":false}" https://localhost/api/1.0/system/password

Managing metrics

Delete data

Metrics can be deleted from Prometheus using the admin API, which is disabled by default for security reasons.

  1. Enable Prometheus Admin API

    • Login to NAbox using SSH and edit /usr/local/nabox/docker-compose/docker-compose.yaml
    • Around line 228, right after - --storage.tsdb.retention.time=2y add - --web.enable-admin-api
    prometheus:                                                                                                                                                                                       
    image: prom/prometheus:latest                                        
    container_name: prometheus                             
    hostname: prometheus                                           
    command:                                                                                        
    - --config.file=/etc/prometheus/prometheus.yml
    - --web.route-prefix=/        
    - --web.external-url=http://example.com/prometheus
    - --storage.tsdb.retention.time=2y                                
    - --web.enable-admin-api

  2. Restart Prometheus

    • dc up -d

  3. Delete metrics for a given cluster 

    # Delete metrics for cluster2
curl -k -L -X POST -g \
  'https://localhost/prometheus/api/v1/admin/tsdb/delete_series?match[]={cluster="cluster2"}'

# Purge data from disk
curl -k -L -X POST http://localhost/prometheus/api/v1/admin/tsdb/clean_tombstones

  4. Restore Prometheus configuration

    • Don't forget to remove --web.enable-admin-apifrom arguments in docker-compose.yaml and run dc up -d again.

If you want to manually delete metrics from old systems or just because you need the space, you can safely remove any file or directory in the /opt/graphite/storage/whisper/ directory. Anything you remove here will then disappear from grafana itself, there is no other database or registry. The basic structure is :

/data/graphite/storage/whisper/netapp/perf/<Site>/<Cluster>/node/<Node> /data/graphite/storage/whisper/netapp/perf/<Site>/<Cluster>/svm/<SVM> /data/graphite/storage/whisper/netapp/capacity/<Site>/<Cluster>/node/<Node> /data/graphite/storage/whisper/netapp/capacity/<Site>/<Cluster>/svm/<SVM>

Inside these various directories you will find sub-directories for volumes, lifs, aggregates, etc... You can use rm -rf to remove them.

Delete old data

Metrics can be deleted from Prometheus using the admin API, which is disabled by default for security reasons.

  1. Enable Prometheus Admin API

    • Login to NAbox using SSH and edit /usr/local/nabox/docker-compose/docker-compose.yaml
    • Around line 228, right after - --storage.tsdb.retention.time=2y add - --web.enable-admin-api
    prometheus:                                                                                                                                                                                       
    image: prom/prometheus:latest                                        
    container_name: prometheus                             
    hostname: prometheus                                           
    command:                                                                                        
    - --config.file=/etc/prometheus/prometheus.yml
    - --web.route-prefix=/        
    - --web.external-url=http://example.com/prometheus
    - --storage.tsdb.retention.time=2y                                
    - --web.enable-admin-api

  2. Restart Prometheus

    • dc up -d

  3. Delete metrics older than 6 months

    # Delete metrics older than 6 months (180 days)
DAYS=180
curl -k -L -X POST -g \
  'https://localhost/prometheus/api/v1/admin/tsdb/delete_series?match[]={job="harvest2"}&end='$(( `date +%s` - $DAYS * 3600 * 24 ))

# Purge data from disk
curl -k -L -X POST http://localhost/prometheus/api/v1/admin/tsdb/clean_tombstones

  4. Restore Prometheus configuration

    • Don't forget to remove --web.enable-admin-apifrom arguments in docker-compose.yaml and run dc up -d again.

To manually purge old metrics, you can run these command lines directly :

find /data/graphite/storage/whisper -type f -mtime +120 -name \*.wsp -delete
find /data/graphite/storage/whisper -depth -type d -empty -delete

Change default retention

If you find that 2 years of data retention is not appropriate in your environment, you can override the default by editing /usr/local/nabox/docker-compose/docker-compose.override.yaml and implement the following block :

services:
  prometheus:
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --web.route-prefix=/
      - --web.external-url=http://example.com/prometheus
      - --storage.tsdb.retention.time=1y
      - ${WEB_ENABLE_ADMIN_API:---log.level=info}

After the change, you have to restart prometheus :

# dc up -d

Customize Harvest 2

Note

This is a NAbox 3.1.1 feature

Customizing Harvest 2 configuration can be a little bit tricky, and is documented here.

That documentation is mostly valid for NAbox, except for the poller custom configuraton file.

NAbox needs to control the content of custom.yaml because it implements its own set of rules for the default ignore list, so you cannot modify that file (and if you do, it will be overwritten!).

  1. You have to implement customizations in your own file in /opt/harvest2-conf/conf/<poller>/ with a name that starts with custom_, i.e. custom_mycustom_conf.yaml.
  2. The object template file parameter must be a list. So instead of doing this like Harvest documentation shows :
objects:
  Aggregate: custom_aggr.yaml

you do this :

objects:
  Aggregate:
    - custom_aggr.yaml

That way the different custom_* will be nicely merged together with the stock files provided with NAbox.

Exclude volumes

To ignore volumes that you don't want to be collected by Harvest performance and/or capacity pollers, you can follow these steps :

  1. Create /opt/harvest2-conf/conf/zapiperf/custom_my_ignore_list.yaml

    objects:
  Volume:
    - custom_my_ignore_list_template.yaml

  2. Create /opt/harvest2-conf/conf/zapiperf/cdot/9.8.0/custom_my_ignore_list_template.yaml

    plugins:
  LabelAgent:
    exclude_regex:
      - volume `Test_volume.*`
      - volume `Temp_volume.*`

  3. We need to do the same for Zapi poller. Create /opt/harvest2-conf/conf/zapi/custom_my_ignore_list.yaml

    objects:
  Volume:
    - custom_my_ignore_list_template.yaml

  4. Create /opt/harvest2-conf/conf/zapi/cdot/9.8.0/custom_my_ignore_list_template.yaml

    plugins:
  LabelAgent:
    exclude_regex:
      - volume `Test_volume.*`
      - volume `Temp_volume.*`

  5. Restart Harvest

    dc restart nabox-harvest2

Customize Grafana

Configure email server

  1. Edit /usr/local/nabox/docker-compose/docker-compose.override.yaml

  2. Configure custom environment variables for grafanaservice. 

    services:
  grafana:
    environment:
      - GF_SMTP_ENABLED=true
      - GF_SMTP_HOST=mysmtpserver.mydomain.com:25
      - GF_SMTP_FROM_ADDRESS=grafana@mydomain.com
      - GF_SERVER_DOMAIN=nabox.mydomain.com
      - GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s/grafana/
      - GF_SERVER_SERVE_FROM_SUB_PATH=true
  3. Restart Grafana 
    dc up -d grafana

Working with Certificate Signing Request

If you need to provide an enterprise certificate signed by your own CA, you can do so using NAbox REST API to issue a valid CSR.

Here is the procedure :

  1. Prepare JSON file for CSR informations 
    {
  "cn": "nabox.home.lab",
  "country": "FR",
  "email": "nabox@company.com",
  "locality": "Paris",
  "organization": "NetApp",
  "organizationalUnit": "Monitoring",
  "sans": [
    "nabox.home.lab",
    "10.1.0.12"
  ],
  "state": "IdF"
}
  2. Send CSR request to NAbox 
    curl -X POST -u admin:Netapp01 https://[nabox]/api/1.0/ssl/csr -H "Content-type: application/json" -d @csr.txt
{
  "status": 200
}
  3. Get the CSR 
    curl -X GET -u admin:Netapp01 https://[nabox]/api/1.0/ssl/csr -H "Content-type: application/json"
{
  "base64": "-----BEGIN CERTIFICATE REQUEST-----\n[...]\n-----END CERTIFICATE REQUEST-----\n",
  "cn": "nabox.home.lab",
  "country": "FR",
  "email": "nabox@company.com",
  "locality": "Paris",
  "organization": "NetApp",
  "organizationalUnit": "Monitoring",
  "sans": [
    "nabox.home.lab",
    "10.1.0.12"
  ],
  "state": "IdF"
}
    You can directly extract Base64 encoded CSR using jq cli : 
    curl -s -X GET -u admin:Netapp01 https://[nabox]/api/1.0/ssl/csr -H "Content-type: application/json"|jq -r .base64
-----BEGIN CERTIFICATE REQUEST-----
MIIDBDCCAewCAQAwgYwxCzAJBgNVBAYTAkZSMSAwHgYJKoZIhvcNAQkBFhFuYWJv
eEBjb21wYW55LmNvbTEMMAoGA1UECAwDSWRGMQ4wDAYDVQQHDAVQYXJpczEPMA0G
A1UECgwGTmV0QXBwMRMwEQYDVQQLDApNb25pdG9yaW5nMRcwFQYDVQQDDA5uYWJv
eC5ob21lLmxhYjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJ0+CVbE
n0kcFq4BFIRwIE6kFfpTwMlNITdRQf2wggzuH0BaTWfqvcyRrIV5OZ+hGzVGLOgo
1mJ99VMrzsHGZfonkLbbucgRggM46D7DmTUatdTFkuRIp5luDrMYhGSuWvf8U87A
VFSCcpdiMUdJtwvzxfmhIMtac78kkrFmyB/y5ISWnpWFeTvF8nVZiHT3S+cZpeS/
cfR2x2VTXoi9UKTVZz8yFU6j4He4h3D5/qXxtNWapoIWUmHBmRHzlCFrj3dWlB2p
lgqJgIyafJ+oQVLXwxCoBMUcGyV+50s6a7ClO95rbYj8GpbvbaO6Gmp0tREFm0G7
u9cSjXzPPz1GsdUCAwEAAaAyMDAGCSqGSIb3DQEJDjEjMCEwHwYDVR0RBBgwFoIO
bmFib3guaG9tZS5sYWKHBAoBAAwwDQYJKoZIhvcNAQELBQADggEBACX0JtchfrDA
t83n6ELbwaYH1iHuNhDg4m85EnMrAID83EwqkzjREwEysGJ6VWAgV8A5A04IyWSB
tuh35V32WeUuc/cn9N5cMVj0GeuwH6iukZQWnmgF8zSFNfr4jE9vrCs2z1amT1OO
HLsFSKISSexoxuBlHroO6B7TUVmg6hVU0i1Qr9rhUfUprdI6SB21HZRHNVuLpT/N
wO+oUH8LAKpGuv7zhg1hPJjMekie/IIra8TkPZJ1TEJhvHQa70Wxcm7Qyzbk9dqG
CHq5tz0avLkcGQE4hlAWpquldd4ymQoh3l5Lz1dbMGTRs29h72oUh0W81UIu+4g6
hJCf25Wozro=
-----END CERTIFICATE REQUEST-----
  4. Send the CSR for signing to your CA
  5. Paste provided certificate into NAbox UI, followed by the intermediate issuer if necessary, without specifying a private key SSL Settings

Allow a few seconds for NAbox to restart web service and it should present the configured certificate.