Troubleshooting
Web Interface Errors and Probable Issues
1. The error occurs in the web interface/Case 1
Error: An Internal Error Has Occurred. Please check that the required services are running.
Probably the mysql server is off/crash. Check mysql server using ssh with the following command:
systemctl status mysql.service
If the service does not look like it's running:
mysql.service - LSB: Start and stop the mysql database server daemon
Loaded: loaded (/etc/init.d/mysql)
Active: inactive (dead) since Mon 2016-09-12 09:37:28 EEST; 1min 43s ago
Process: 15510 ExecStop=/etc/init.d/mysql stop (code=exited, status=0/SUCCESS)
Process: 548 ExecStart=/etc/init.d/mysql start (code=exited, status=0/SUCCESS)
The problem is resolved with the mysql.service restart system, using the following command:
systemctl restart mysql.service
Check using the following command:
systemctl status mysql.service
The result should look like this:
mysql.service - LSB: Start and stop the mysql database server daemon
Loaded: loaded (/etc/init.d/mysql)
Active: active (running) since Mon 2016-09-12 09:40:48 EEST; 2s ago
Process: 15510 ExecStop=/etc/init.d/mysql stop (code=exited, status=0/SUCCESS)
Process: 15959 ExecStart=/etc/init.d/mysql start (code=exited, status=0/SUCCESS)
2. The error occurs in the web interface/Case 2
Error: Connecting to the main datastore. Are all the services running? Connection refused
Probably the mysql server is off/crash. Check mysql server using ssh with the following command:
systemctl status mysql.service
If the service does not look like it's running :
mysql.service - LSB: Start and stop the mysql database server daemon
Loaded: loaded (/etc/init.d/mysql)
Active: inactive (dead) since Mon 2016-09-12 09:37:28 EEST; 1min 43s ago
Process: 15510 ExecStop=/etc/init.d/mysql stop (code=exited, status=0/SUCCESS)
Process: 548 ExecStart=/etc/init.d/mysql start (code=exited, status=0/SUCCESS)
The problem is resolved with the mysql.service restart system, using the following command:
systemctl restart mysql.service
Check using the following command:
systemctl status mysql.service
The result should look like this:
mysql.service - LSB: Start and stop the mysql database server daemon
Loaded: loaded (/etc/init.d/mysql)
Active: active (running) since Mon 2016-09-12 09:40:48 EEST; 2s ago
Process: 15510 ExecStop=/etc/init.d/mysql stop (code=exited, status=0/SUCCESS)
Process: 15959 ExecStart=/etc/init.d/mysql start (code=exited, status=0/SUCCES
Or probably elasticsearch is off/crash. Check elasticsearch service using ssh with the following command:
systemctl status elasticsearch.service
If the service does not look like it's running :
elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled)
Active: inactive (dead) since Mon 2016-09-12 09:41:48 EEST; 2min 5s ago
Docs: http://www.elastic.co
Process: 608 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_HOME} -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR} -Des.default.path.conf=${CONF_DIR} (code=exited, status=143)
Process: 517 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 608 (code=exited, status=143)
The problem is resolved with the elasticsearch.service restart system, using the following command:
systemctl restart elasticsearch.service
Check using the following command:
systemctl status elasticsearch.service
The result should look like this:
elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled)
Active: active (running) since Mon 2016-09-12 09:44:28 EEST; 2s ago
Docs: http://www.elastic.co
Process: 16854 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 16856 (java)
CGroup: /system.slice/elasticsearch.service
└─16856 /usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX...
3. The error occurs in the web interface/Case 3
Error: Connecting to the main datastore. Are all the services running? ElasticSearch Error: Error
Probably elasticsearch is off/crash. Check elasticsearch service using ssh with the following command:
systemctl status elasticsearch.service
If the service does not look like it's running:
elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled)
Active: inactive (dead) since Mon 2016-09-12 09:41:48 EEST; 2min 5s ago
Docs: http://www.elastic.co
Process: 608 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_HOME} -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR} -Des.default.path.conf=${CONF_DIR} (code=exited, status=143)
Process: 517 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 608 (code=exited, status=143)
The problem is resolved with the elasticsearch.service restart system, using the following command:
systemctl restart elasticsearch.service
Check using the following command:
systemctl status elasticsearch.service
The result should look like this:
elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled)
Active: active (running) since Mon 2016-09-12 09:44:28 EEST; 2s ago
Docs: http://www.elastic.co
Process: 16854 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 16856 (java)
CGroup: /system.slice/elasticsearch.service
└─16856 /usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX...
4. Mysql service off
The error which appears in the “putty” with the corresponding host name (or ip address) and the port and also the connection type (SSH) using the command:
systemctl status mysql.service
And the error following commands:
Now if we go into the web application this will be the error that will occur:
Error: An Internal Error Has Occurred. Please check the that required services are running
The resolving method is to restart mysql service by following the command:
systemctl restart mysql.service
Check using the following command:
systemctl status mysql.service
And the result of the command is the following:
5. Elasticsearch service off
The error which appears in the “putty” with the corresponding host name (or ip address) and the port and also the connection type (SSH) using the command:
systemctl status elasticsearch.service
And the error following commands:
Now if we go into cerebro web plugin (http://CyberquestIP:9000) will show the following error:
Or if we go into the web application this will be the error that will occur and there is no data available:
The resolving method is to restart the service by following the command:
systemctl restart elasticsearch.service
Check using the following command:
systemctl status elasticsearch.service
And the result of the command is the following:
6. Rabbitmq-server service stopped
The error which appears in the “putty” with the corresponding host name (or ip address) and the port and also the connection type (SSH) using the command:
systemctl status rabbitmq-server.service
And the error following commands:
Now if we go into rabbitmq web plugin (http://CyberquestIP:15672) does not work:
The resolving method is to restart the service by following the commands:
systemctl restart rabbitmq-server.service
AND
systemctl restart data-acquisition.service
Check using the following command:
systemctl status rabbitmq-server.service
And the result of the command is the following:
7. Nginx.service service stopped
The error which appears in the “putty” with the corresponding host name (or ip address) and the port and also the connection type (SSH) using the command:
systemctl status nginx.service
And the error following commands:
Or if we go into the web application this will be the error that will occur:
The resolving method is to restart the service by following the command:
systemctl restart nginx.service
Check using the following command:
systemctl status nginx.service
The result of the command is the following:
8. Php7.4-fpm.service service stopped
The error which appears in the “putty” with the corresponding host name (or ip address) and the port and also the connection type (SSH) using the command:
systemctl status php7.4-fpm.service
And the error following commands:
Or if we go into the web application this will be the error that will occur:
The resolving method is to restart the service by following the command:
systemctl restart php7.4-fpm.service
Check using the following command:
systemctl status status php7.4-fpm.service
The result of the command is the following:
9. Data-storage.service service stopped
The error which appears in the “putty” with the corresponding host name (or ip address) and the port and also the connection type (SSH) using the command:
systemctl status data-storage.service
And the error following commands:
Events blocked on the queue in Data Storage on RabbitMQ(http://CyberquestIP:15672)
The resolving method is to restart the service by following the command:
systemctl restart data-storage.service
Check using the following command:
systemctl status data-storage.service
The result of the command is the following:
10. Data-acquisition.service service stopped
The error which appears in the “putty” with the corresponding host name (or ip address) and the port and also the connection type (SSH) using the command:
systemctl status data-acquisition.service
And the error following commands:
Events blocked on the queue in data-acquisition.service on RabbitMQ (http://CyberquestIP:15672)
The resolving method is to restart the service by following the command:
systemctl restart data-acquisition.service
Check using the following command:
systemctl status data-acquisition.service
The result of the command is the following:
Or if we go into the web application this will be the error that will occur:
ERROR: Connecting to the main datastore. Are all the services running? Index “el_logs_current” is missing
Now if we go into cerebro web plugin (http://CyberquestIP:9000) will show the following error:
The resolving method is to restart the service by following the command:
systemctl restart data-acquisition.service
11. Rsyslog service stopped (self-audit events)
The error which appears in the “putty” with the corresponding host name (or ip address) and the port and also the connection type (SSH) using the command:
/etc/init.d/rsyslog status
And the error following commands:
The resolving method is to restart the service by following the command:
/etc/init.d/rsyslog restart
And the result of the command is the following:
/etc/init.d/rsyslog status
Or another problem can be found in rsyslog.conf file:
nano /etc/rsyslog.conf
Scroll down until you reach the end of the file and check if it matches the desired ip (" *. * " - this symbol represents all types of events)
12. RabbitMQ error
When this error occurs, we don’t find new information in the web application. We find this error in CYBERQUEST logs. Using the baretail program we open the file(agent.txt) and we can view the error log.
To solve the problem we have to stop the CYBERQUEST Agent and restart the RabbitMQ service. Follow the instructions for more details:
A. Stop CYBERQUEST Agent
On the Windows machine where the CYBERQUEST Agent is installed, open Windows Services and stop the CYBERQUEST Agent service.
To check if the CYBERQUEST Agent has stopped, open Task Manager > Details and wait until the Agent .exe process disappears from Task Manager.
B. Restart RabbitMQ
We connect to the CYBERQUEST server via ssh (e.g. ssh user@ip) and restart RabbitMQ using the following command:
systemctl restart rabbitmq-server.service
C. Start CYBERQUEST Agent
On the Windows machine where the CYBERQUEST Agent is installed, open Windows Services and start the CYBERQUEST Agent service.
To verify that the CYBERQUEST Agent has started, open Task Manager> Details and wait until the Agent.exe process appears in the Task Manager.